Abstract
High-performance computing (HPC) solutions require high bandwidth, low latency components with CPU offloads to get the highest server efficiency and application productivity. The Mellanox ConnectX-3 FDR VPI InfiniBand/Ethernet and 10 Gb Ethernet adapters for IBM® System x® deliver the I/O performance that meets these requirements. This IBM Redbooks® Product Guide provides an introduction to these adapters.
Mellanox ConnectX-3 delivers low latency, high bandwidth, and computing efficiency for performance-driven server and storage clustering applications. Efficient computing is achieved by offloading from the CPU routine activities, which makes more processor power available for the application. Network protocol processing and data movement impacts, such as InfiniBand RDMA and Send/Receive semantics, are completed in the adapter without CPU intervention. The ConnectX-3 advanced acceleration technology enables higher cluster efficiency and scalability of up to tens of thousands of nodes.
Contents
High-performance computing (HPC) solutions require high bandwidth, low latency components with CPU offloads to get the highest server efficiency and application productivity. The Mellanox ConnectX-3 FDR VPI InfiniBand/Ethernet and 10 Gb Ethernet adapters for IBM® System x® deliver the I/O performance that meets these requirements.
Mellanox ConnectX-3 delivers low latency, high bandwidth, and computing efficiency for performance-driven server and storage clustering applications. Efficient computing is achieved by offloading from the CPU routine activities, which makes more processor power available for the application. Network protocol processing and data movement impacts, such as InfiniBand RDMA and Send/Receive semantics, are completed in the adapter without CPU intervention. The ConnectX-3 advanced acceleration technology enables higher cluster efficiency and scalability of up to tens of thousands of nodes.
Figure 1. Mellanox Connect X-3 10GbE Adapter for IBM System x (3U bracket shown)
Did you know?
The Mellanox ConnectX-3 FDR VPI IB/E Adapter supports Virtual Protocol Interconnect, which enables you to have either two ports of InfiniBand FDR (56 Gbps) or two ports of 40 Gb Ethernet, or one port of each.
ConnectX-3 with Virtual Intelligent Queuing (Virtual-IQ) technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server. I/O virtualization with ConnectX-3 gives data center managers better server utilization and LAN and SAN unification while reducing cost, power, and cable complexity.
Part number information
Table 1 shows the part numbers and feature codes for the adapters.
Table 1. Ordering part numbers and feature codes
| Part number | Feature code | Description |
| 00D9690 | A3PM | Mellanox Connect X-3 10GbE Adapter for IBM System x |
| 00D9550 | A3PN | Mellanox ConnectX-3 FDR VPI IB/E Adapter for IBM System x |
The part numbers include the following:
- One adapter with a 2U bracket attached
- 3U bracket included in the box
- Quick installation guide
- Documentation CD
- Warranty Flyer
- IBM Important Notices Flyer
The 10GbE Adapter (00D9690) supports the direct-attach copper (DAC) twin-ax cables, transceivers, and optical cables that are listed in the following table.
Table 2. Supported transceivers and DAC cables - Mellanox Connect X-3 10GbE Adapter
| Part number | Feature code | Description |
| Direct Attach Copper (DAC) cables | ||
| 00D6288 | A3RG | .5 m IBM Passive DAC SFP+ Cable |
| 90Y9427 | A1PH | 1 m IBM Passive DAC SFP+ Cable |
| 90Y9430 | A1PJ | 3 m IBM Passive DAC SFP+ Cable |
| 90Y9433 | A1PK | 5 m IBM Passive DAC SFP+ Cable |
| 00D6151 | A3RH | 7 m IBM Passive DAC SFP+ Cable |
| SFP+ Transceivers | ||
| 44W4408 | 4942 | IBM 10Gb SFP+ SR Optical Transceiver (BN-CKM-SP-SR) |
| 46C3447 | 5053 | IBM SFP+ SR Transceiver |
| 45W2411 |  | SFP+ Transceiver 10 Gbps SR (IB-000180) |
| 49Y4216 | 0069 | Brocade 10Gb SFP+ SR Optical Transceiver |
| 49Y4218 | 0064 | QLogic 10Gb SFP+ SR Optical Transceiver |
| Optical Cables | ||
| 88Y6851 | A1DS | 1 m LC-LC Fiber Cable (networking) - Optical |
| 88Y6854 | A1DT | 5 m LC-LC Fiber Cable (networking)- Optical |
| 88Y6857 | A1DU | 25 m LC-LC Fiber Cable (networking) - Optical |
The FDR VPI IB/E Adapter (00D9550) supports the direct-attach copper (DAC) twin-ax cables, transceivers, and optical cables that are listed in the following table.
Note: InfiniBand cables and the 10 Gb Ethernet cable should be sourced from Mellanox.
Table 3. Supported transceivers and DAC cables - Mellanox ConnectX-3 FDR VPI IB/E Adapter
| Part number | Feature code | Description |
| Direct attach copper (DAC) cables - InfiniBand | ||
| MC2207130-001 | None | 1 m Mellanox QSFP Passive Copper FDR14 InfiniBand Cable |
| MC2207128-003 | None | 3 m Mellanox QSFP Passive Copper FDR14 InfiniBand Cable |
| Optical cables - InfiniBand | ||
| MC2207310-003 | None | 3 m Mellanox QSFP Optical FDR14 InfiniBand Cable |
| MC2207310-005 | None | 5 m Mellanox QSFP Optical FDR14 InfiniBand Cable |
| MC2207310-010 | None | 10 m Mellanox QSFP Optical FDR14 InfiniBand Cable |
| MC2207310-015 | None | 15 m Mellanox QSFP Optical FDR14 InfiniBand Cable |
| MC2207310-020 | None | 20 m Mellanox QSFP Optical FDR14 InfiniBand Cable |
| MC2207310-030 | None | 30 m Mellanox QSFP Optical FDR14 InfiniBand Cable |
| 10Gb Ethernet (SFP+) | ||
| MC2309130-003 | None | 3 m QSFP+ to SFP+ passive copper cables |
| 40Gb Ethernet (QSFP) - 40GbE copper uses the QSFP+ to QSFP+ cables directly. | ||
| 74Y6074 / 49Y7890 | A1DP | 1 m IBM QSFP+ to QSFP+ Cable |
| 74Y6075 / 49Y7891 | A1DQ | 3 m IBM QSFP+ to QSFP+ Cable |
| 40Gb Ethernet (QSFP) - 40GbE optical uses QSFP+ transceiver with MTP optical cables | ||
| 49Y7884 | A1DR | IBM QSFP+ 40GBASE-SR4 Transceiver |
| 41V2458 / 90Y3519 | A1MM | 10 m IBM QSFP+ MTP Optical cable |
| 45D6369 / 90Y3521 | A1MN | 30 m IBM QSFP+ MTP Optical cable |
Figure 2 shows the Mellanox ConnectX-3 FDR VPI IB/E Adapter
Figure 2. Mellanox ConnectX-3 FDR VPI IB/E Adapter for IBM System x (3U bracket shown)
Features
The Mellanox Connect X-3 10GbE Adapter has the following features:
- Two 10 Gigabit Ethernet ports
- Low-profile form factor adapter with 2U bracket (3U bracket available for CTO orders)
- PCI Express 3.0 x8 host-interface (PCIe 2.0 and 1.1 compatible)
- Enables Low Latency RDMA over Ethernet
- TCP/UDP/IP stateless offload in hardware
- Traffic steering across multiple cores
- Intelligent interrupt coalescence
- Advanced Quality of Service and congestion control
- Hardware-based I/O virtualization
- Industry-leading throughput and latency performance
- Software compatible with standard TCP/UDP/IP stacks
The Mellanox ConnectX-3 FDR VPI IB/E Adapter has the following features:
- Dual QSFP ports supporting FDR Ethernet or 40 Gb Ethernet
- Low-profile form factor adapter with 2U bracket (3U bracket available for CTO orders)
- PCI Express 3.0 x8 host-interface (PCIe 2.0 and 1.1 compatible)
- Support for InfiniBand FDR speeds of up to 56 Gbps (auto-negotiation DDR and SDR)
- Support for Virtual Protocol Interconnect (VPI), which enables one adapter for both InfiniBand and 10/40 Gb Ethernet. Supports three configurations:
- 2 ports 56 Gb InfiniBand
- 2 ports 40 Gb Ethernet
- 1 port 56 Gb InfiniBand and 1 port 40 Gb Ethernet
- High performance/low-latency networking
- Sub 1 µs MPI ping latency
- Support for QSFP to SFP+ for 10 GbE support
- Traffic steering across multiple cores
Performance
Based on Mellanox ConnectX-3 technology, these adapters provide a high level of throughput performance for all network environments by removing I/O bottlenecks in mainstream servers that are limiting application performance. With the FDR VPI IB/E Adapter, servers can achieve up to 56 Gbps transmit and receive bandwidth. Hardware-based InfiniBand transport and IP over InfiniBand (IPoIB) stateless off-load engines handle the segmentation, reassembly, and checksum calculations that otherwise burden the host processor.
RDMA over InfiniBand and RDMA over Ethernet further accelerate application run time while reducing CPU utilization. RDMA allows very high-volume transaction-intensive applications typical of HPC and financial market firms, as well as other industries where speed of data delivery is paramount. With the ConnectX-3-based adapter, highly compute-intensive tasks running on hundreds or thousands of multiprocessor nodes, such as climate research, molecular modeling, and physical simulations, can share data and synchronize faster, resulting in shorter run times.
These adapters implement RDMA over Converged Ethernet (RoCE) technology, which delivers low latency and high performance over Ethernet networks. Leveraging Data Center Bridging capabilities, RoCE provides efficient low latency RDMA services over Layer 2 Ethernet. The RoCE software stack maintains existing and future compatibility with bandwidth- and latency-sensitive applications. With link-level interoperability in the existing Ethernet infrastructure, network administrators can use existing data center fabric management solutions.
In data mining or web crawl applications, RDMA provides the needed boost in performance to enable faster search by solving the network latency bottleneck that is associated with I/O cards and the corresponding transport technology in the cloud. Various other applications that benefit from RDMA with ConnectX-3 include Web 2.0 (Content Delivery Network), business intelligence, database transactions, and various cloud computing applications. Mellanox ConnectX-3's low power consumption provides clients with high bandwidth and low latency at the lowest cost of ownership.
Quality of service
Resource allocation per application or per VM is provided by the advanced quality of service (QoS) supported by ConnectX-3. Service levels for multiple traffic types can be assigned on a per flow basis, allowing system administrators to prioritize traffic by application, virtual machine, or protocol. This powerful combination of QoS and prioritization provides the ultimate fine-grained control of traffic, ensuring that applications run smoothly in today’s complex environments.
TCP/UDP/IP acceleration
Applications utilizing TCP/UDP/IP transport can achieve industry leading throughput over InfiniBand or 10 GbE adapters. The hardware-based stateless offload engines in ConnectX-3 reduce the CPU impact of IP packet transport, allowing more processor cycles to work on the application.
Software support
All Mellanox adapter cards are supported by a full suite of drivers for Microsoft Windows, Linux distributions, VMware, and Citrix XENServer. ConnectX-3 adapters support OpenFabrics-based RDMA protocols and software. Stateless offload is fully interoperable with standard TCP/ UDP/IP stacks. ConnectX-2 adapters are compatible with configuration and management tools from OEMs and operating system vendors.
Specifications
InfiniBand specifications (ConnectX-3 FDR VPI IB/E Adapter):
- Supports InfiniBand FDR, FDR10, QDR, DDR, and SDR
- IBTA Specification 1.2.1 compliant
- RDMA, Send/Receive semantics
- Hardware-based congestion control
- 16 million I/O channels
- 256 to 4 KB MTU, 1 GB messages
- Nine virtual lanes: Eight data and one management
Enhanced InfiniBand specifications:
- Hardware-based reliable transport
- Hardware-based reliable multicast
- Extended Reliable Connected transport
- Enhanced Atomic operations
- Fine-grained end-to-end quality of server (QoS)
Ethernet specifications:
- IEEE 802.3ae 10 Gigabit Ethernet
- IEEE 802.3ba 40 Gigabit Ethernet (ConnectX-3 FDR VPI IB/E Adapter)
- IEEE 802.3ad Link Aggregation and Failover
- IEEE 802.1Q, 1p VLAN tags and priority
- IEEE P802.1au D2.0 Congestion Notification
- IEEE P802.1az D0.2 Enhanced Transmission Selection (ETS)
- IEEE P802.1bb D1.0 Priority-based Flow Control
- IEEE 1588 Precision Clock Synchronization
- Multicast
- Jumbo frame support
- 128 MAC/VLAN addresses per port
Hardware-based I/O virtualization:
- Address translation and protection
- Multiple queues per virtual machine
- VMware NetQueue support
Additional CPU offloads:
- TCP/UDP/IP stateless offload
- Intelligent interrupt coalescence
- Compliant with Microsoft RSS and NetDMA
Storage support:
- Fibre Channel over InfiniBand ready
- Fibre Channel over Ethernet ready
Management and tools:
InfiniBand:
- OpenSM
- Interoperable with third-party subnet managers
- Firmware and debug tools (MFT and IBDIAG)
- MIB, MIB-II, MIB-II Extensions, RMON, and RMON 2
- Configuration and diagnostic tools
Protocol support:
- Open MPI, OSU MVAPICH, Intel MPI, MS MPI, and Platform MPI
- TCP/UDP, EoIB, IPoIB, SDP, and RDS
- SRP, iSER, NFS RDMA, FCoIB, and FCoE
- uDAPL
Physical specifications
The adapters have the following dimensions:
- Height: 168 mm (6.60 in)
- Width: 69 mm (2.71 in)
- Depth: 17 mm (0.69 in)
- Weight: 208 g (0.46 lb)
Approximate shipping dimensions:
- Height: 189 mm (7.51 in)
- Width: 90 mm (3.54 in)
- Depth: 38 mm (1.50 in)
- Weight: 450 g (0.99 lb)
Regulatory approvals
- EN55022
- EN55024
- EN60950-1
- EN 61000-3-2
- EN 61000-3-3
- IEC 60950-1
- FCC Part 15 Class A
- UL 60950-1
- CSA C22.2 60950-1-07
- VCCI
- NZ AS3548 / C-tick
- RRL for MIC (KCC)
- BSMI (EMC)
- IECS-003:2004 Issue 4
Operating environment
The adapters are supported in the following environment:
Operating temperature:
- 0 - 55° C (-32 to 131° F) at 0 - 914 m (0 - 3,000 ft)
- 10 - 32° C (50 - 90° F) at 914 to 2133 m (3,000 - 7,000 ft)
Relative humidity: 20% - 80% (noncondensing)
Maximum altitude: 2,133 m (7,000 ft)
Air flow: 200 LFM at 55° C
Power consumption:
- Power consumption (typical): 8.8 W typical (both ports active)
- Power consumption (maximum): 9.4 W maximum for passive cables only,
- 13.4 W maximum for active optical modules
Warranty
One year limited warranty. When installed in an IBM System x server, these cards assume your system’s base warranty and any IBM ServicePac® upgrades.
Supported servers
The adapters are supported in the System x servers that are listed in the following table.
Table 4. Supported System x and IBM iDataPlex® servers (Part 1)
| Part number | Product description | ||||||||||
| 00D9690 | Mellanox Connect X-3 10GbE Adapter | N | N | N | N | Y | N | N | Y | N | N |
| 00D9550 | Mellanox ConnectX-3 FDR VPI IB/E Adapter | N | N | N | N | Y | N | N | Y | N | N |
Table 4. Supported System x and iDataPlex servers (Part 2)
| Part number | Product description | ||||||||||||
| 00D9690 | Mellanox Connect X-3 10GbE Adapter | N | N | Y | N | Y | Y | Y | Y | Y | Y | Y | Y |
| 00D9550 | Mellanox ConnectX-3 FDR VPI IB/E Adapter | N | N | Y | N | Y | Y | Y | Y | Y | Y | Y | Y |
Memory requirements: Ensure that your server has sufficient memory available to the adapters. The first adapter requires 8 GB of RAM in addition to the memory allocated to operating system, applications, and virtual machines. Any additional adapters installed each require 4 GB of RAM.
Supported operating systems
The adapters support the following operating systems:
- SUSE LINUX Enterprise Server 10 for AMD64/EM64T (SP4)
- SUSE LINUX Enterprise Server 11 for AMD64/EM64T (SP2)
- Red Hat Enterprise Linux 5 Server x64 Edition (U8)
- Red Hat Enterprise Linux 6 Server x64 Edition (U3)
- VMware vSphere 5 (U1)
- VMware vSphere 5.1
Note on VMware Support: VMware is supported on Mellanox Connect X-3 10 GbE Adapter for IBM System x (00D9690), and on Mellanox ConnectX-3 FDR VPI IB/E Adapter for IBM System x (00D9550) only in the Ethernet mode.
Related publications
For more information, refer to these documents:
- IBM U.S. Announcement Letter
http://ibm.com/common/ssi/cgi-bin/ssialias?infotype=dd&subtype=ca&&htmlfid=897/ENUS113-008 - IBM System x networking home page
http://ibm.com/systems/x/options/networking/adapters.html - Mellanox User Manual - Mellanox ConnectX-3 FDR VPI IB/E Adapter
http://bit.ly/YJ3R0x - Mellanox User Manual - Mellanox Connect X-3 10GbE Adapter
http://bit.ly/YJ4oQa
Special Notices
This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment.