Lenovo Flex System EN6131 40Gb Ethernet Switch

Product Guide

This document is being moved!
In September 2015, this document will be moved to lenovopress.com and will no longer be available on the IBM Redbooks site. Please bookmark the URL http://lenovopress.com/TIPS0911 to access this specific document both today and after the migration in September.

Learn more

Abstract

The Lenovo Flex System™ EN6131 40Gb Ethernet Switch in conjunction with the EN6132 40Gb Ethernet Adapter offer the performance that you need to support clustered databases, parallel processing, transactional services, and high-performance embedded I/O applications, which reduces task completion time and lowers the cost per operation. This switch offers 14 internal and 18 external 40 Gb Ethernet ports that enable a non-blocking network design. It supports all Layer 2 functions so servers can communicate within the chassis without going to a Top-of-Rack (ToR) switch, which helps improve performance and latency.

Changes in the July 20 update:
* Updated latency

Contents


Table of contents


Introduction

The Lenovo Flex System™ EN6131 40Gb Ethernet Switch in conjunction with the EN6132 40Gb Ethernet Adapter offer the performance that you need to support clustered databases, parallel processing, transactional services, and high-performance embedded I/O applications, which reduces task completion time and lowers the cost per operation.

This switch offers 14 internal and 18 external 40 Gb Ethernet ports that enable a non-blocking network design. It supports all Layer 2 functions so servers can communicate within the chassis without going to a Top-of-Rack (ToR) switch, which helps improve performance and latency.

The EN6131 40Gb Ethernet Switch is shown in Figure 1.

IBM Flex System EN6131 40Gb Ethernet Switch

Figure 1. Lenovo Flex System EN6131 40Gb Ethernet Switch

Did you know?

The 40 Gb Ethernet solution that is offered by Lenovo Flex System can deploy more workloads per server without running into I/O bottlenecks. If there are failures or server maintenance, clients can also move their virtual machines much faster by using 40 Gb interconnects within the chassis.

The 40 GbE switch and adapter are designed for low latency, high bandwidth, and computing efficiency for performance-driven server and storage clustering applications. They provide extreme scalability for low-latency clustered solutions with reduced packet hops.

The Flex System 40 GbE solution offers the highest bandwidth without adding any significant power impact. It can also help increase the system usage and decrease the number of network ports for further cost savings.

Part number information

Table 1 shows the part numbers for ordering the EN6131 switch.

Table 1. Part numbers and feature codes for ordering

DescriptionPart numberFeature code
Lenovo Flex System EN6131 40Gb Ethernet Switch90Y9346A3HJ

The part number for the EN6131 switch includes the following items:
  • One Lenovo Flex System EN6131 40Gb Ethernet Switch
  • Technical Update Flyer
  • Important Notices Flyer
  • Warranty Flyer
  • Documentation CD-ROM

Note: No QSFP+ (quad small form-factor pluggable plus) transceivers or cables are included with the switch. They must be ordered separately (see Table 2).

The switch does not include a serial management cable. However, Flex System Management Serial Access Cable, 90Y9338, is supported and contains two cables, a mini-USB-to-RJ45 serial cable and a mini-USB-to-DB9 serial cable, either of which can be used to connect to the switch module locally for configuration tasks and firmware updates.

Supported cables and transceivers

Table 2 lists the supported cables and transceivers.

Table 2. Supported transceivers and direct-attach cables

DescriptionPart numberFeature codeMaximum quantity supported
Serial console cables
Flex System Management Serial Access Cable Kit90Y9338A2RR1
QSFP+ transceiver and cables - 40 GbE
Lenovo 40GBASE-SR4 QSFP+ Transceiver49Y7884A1DR18
Optical cables for 40 GbE QSFP+ SR4 transceivers
Lenovo 10m QSFP+ MTP-MTP OM3 MMF Cable90Y3519A1MM18
Lenovo 30m QSFP+ MTP-MTP OM3 MMF Cable90Y3521A1MN18
Lenovo 10m QSFP+ MTP-MTP OM3 MMF Cable (replaces 90Y3519)00VX003AT2U18
Lenovo 30m QSFP+ MTP-MTP OM3 MMF Cable (replaces 90Y3521)00VX005AT2V18
QSFP+ direct-attach cables - 40 GbE
3m FDR InfiniBand Cable90Y3470A22718
Lenovo 3m Passive QSFP+ DAC Cable49Y7891A1DQ18
Lenovo 5m Passive QSFP+ DAC Cable00D5810A2X818
Lenovo 7m Passive QSFP+ DAC Cable00D5813A2X918

Benefits

The EN6131 40Gb Ethernet Switch offers the following benefits:

  • End-to-end 40 Gb connectivity with non-blocking throughput and 230 ns latency
  • Ideal for clients running Ethernet infrastructure in high speed trading, Web 2.0, virtualization, and cloud computing
  • Enables clients to maximize compute usage by removing an I/O bottleneck
  • Low power consumption with less than 0.1 watts per Gbps
  • Can linearly scale to larger node counts to create a low latency clustered solution and reduce packet hops
  • Reduces cabling up to four times, which simplifies deployment and maintenance

Features and specifications

The EN6131 40Gb Ethernet Switch has the following features and specifications:

  • MLNX-OS operating system
  • Internal ports
    • Fourteen internal full-duplex 40 Gigabit ports (10, 20, or 40 Gbps auto-negotiation).
    • One internal full-duplex 1 GbE port that is connected to the chassis management module.
  • External ports
    • Eighteen ports for 40 Gb Ethernet QSFP+ transceivers or QSFP+ DAC cables (10, 20, or 40 Gbps auto-negotiation). QSFP+ modules and DAC cables are not included and must be purchased separately.
    • One external 1 GbE port with RJ-45 connector for switch configuration and management.
    • One RS-232 serial port (mini-USB connector) that provides an additional means to configure the switch module.
  • Scalability and performance
    • 40 Gb Ethernet ports for extreme bandwidth and performance.
    • Non-blocking architecture with wire-speed forwarding of traffic and an aggregated throughput of 1.44 Tbps.
    • Support for up to 48,000 unicast and up to 16,000 multicast media access control (MAC) addresses per subnet.
    • Static and LACP (IEEE 802.3ad) link aggregation, up to 720 Gb of total uplink bandwidth per switch, up to 36 link aggregation groups (LAGs), and up to 16 ports per LAG.
    • Support for jumbo frames (up to 9,216 bytes).
    • Broadcast/multicast storm control.
    • IGMP snooping to limit flooding of IP multicast traffic.
    • Fast port forwarding and fast uplink convergence for rapid STP convergence.
  • Availability and redundancy
    • IEEE 802.1D STP for providing L2 redundancy.
    • IEEE 802.1w Rapid STP (RSTP) provides rapid STP convergence for critical delay-sensitive traffic such as voice or video.
  • VLAN support
    • Up to 4094 VLANs are supported per switch, with VLAN numbers 1 - 4094.
    • 802.1Q VLAN tagging support on all ports.
  • Security
    • Up to 24,000 rules with VLAN-based, MAC-based, protocol-based, and IP-based access control lists (ACLs).
    • User access control.
      • Multiple user IDs and passwords
      • RADIUS, TACACS+, and LDAP authentication and authorization
  • Quality of service (QoS)
    • Support for IEEE 802.1p traffic processing.
    • Traffic shaping that is based on defined policies.
    • Four Weighted Round Robin (WRR) priority queues per port for processing qualified traffic.
    • Priority-Based Flow Control (PFC) (IEEE 802.1Qbb) extends 802.3x standard flow control to allow the switch to pause traffic based on the 802.1p priority value in each packet’s VLAN tag.
    • Enhanced Transmission Selection (ETS) (IEEE 802.1Qaz) provides a method for allocating link bandwidth based on the 802.1p priority value in each packet’s VLAN tag.
  • Manageability
    • IPv4 and IPv6 host management.
    • Simple Network Management Protocol (SNMP V1, V2, and V3).
    • WebUI graphical user interface.
    • Industry standard command-line interface (IS-CLI) through Telnet, SSH, and serial port.
    • Link Layer Discovery Protocol (LLDP) to advertise the device's identity, capabilities, and neighbors.
    • Firmware image update (TFTP, FTP, and SCP).
    • Network Time Protocol (NTP) for clock synchronization.
  • Monitoring
    • Switch LEDs for external port status and switch module status indication.
    • Port mirroring for analyzing network traffic passing through the switch.
    • Change tracking and remote logging with the syslog feature.
    • Support for sFLOW agent for monitoring traffic in data networks (separate sFLOW collector/analyzer is required elsewhere).
    • POST diagnostic tests.

Standards supported

The switch supports the following Ethernet standards:

  • IEEE 802.1AB Link Layer Discovery Protocol
  • IEEE 802.1D Spanning Tree Protocol (STP)
  • IEEE 802.1p Class of Service (CoS) prioritization
  • IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled)
  • IEEE 802.1Qbb Priority-Based Flow Control (PFC)
  • IEEE 802.1Qaz Enhanced Transmission Selection (ETS)
  • IEEE 802.1w Rapid STP (RSTP)
  • IEEE 802.3ab 1000BASE-T copper twisted pair Gigabit Ethernet
  • IEEE 802.3ad Link Aggregation Control Protocol
  • IEEE 802.3ba 40GBASE-SR4 short range fiber optics 40 Gb Ethernet
  • IEEE 802.3ba 40GBASE-CR4 copper 40 Gb Ethernet
  • IEEE 802.3u 100BASE-TX Fast Ethernet
  • IEEE 802.3x Full-duplex Flow Control

Supported chassis and adapters

The EN6131 switches are installed in the I/O module bays in the rear of the Flex System chassis, as shown in Figure 2. Switches are normally installed in pairs because ports on the I/O adapters that are installed in the compute nodes are routed to two I/O bays for redundancy and performance.

Location of the I/O module bays in the Flex System chassis
Figure 2. Location of the I/O module bays in the Flex System chassis

The Flex System EN6131 40Gb Ethernet Switch can be installed in bays 1, 2, 3, and 4 of the Enterprise Chassis. A supported Ethernet adapter must be installed in the corresponding slot of the compute node.

In the compute nodes that have an integrated dual-port 10 GbE network interface controller (NIC), NIC ports are routed to bays 1 and 2 with a specialized periscope connector, and the adapter is not required. However, when needed, the periscope connector can be replaced with the adapter. In such a case, the integrated NIC is disabled.

4-port 10 GbE adapters: With 4-port 10 GbE adapters, only up to two adapter ports can be used with the EN6131 switches (one port per switch).

8-port 10 GbE adapters: 8-port 10 GbE adapters are not supported by the EN6131. See Table 5 for a list of supported adapters.

Table 3 shows compatibility information for the EN6131 and Flex System chassis.

Table 3. Flex System chassis compatibility

DescriptionPart numberEnterprise Chassis with CMMEnterprise Chassis with CMM2Carrier-Grade Chassis with CMM2
Lenovo Flex System EN6131 40Gb Ethernet Switch90Y9346
Yes
Yes
No

The connections between the adapters that are installed in the compute nodes to the EN6131 switches that are installed in the chassis I/O bays are shown in Figure 3. The figure also shows both half-wide compute nodes, such as the x240 with two adapters, and full-wide compute nodes, such as the x440 with four adapters.

Logical layout of the interconnects between I/O adapters and EN6131 modules
Figure 3. Logical layout of the interconnects between I/O adapters and EN6131 modules

Table 4 shows the connections between the adapters that are installed in the compute nodes to the EN6131 switches that are installed in the chassis.

Table 4. Adapter to EN6131 I/O bay correspondence
I/O adapter slot in the serverPort on the adapterCorresponding I/O module bay in the chassis
Bay 1Bay 2Bay 3Bay 4
Slot 1Port 1Yes
Port 2Yes
Slot 2Port 1Yes
Port 2Yes
Slot 3
(full-wide compute nodes only)
Port 1Yes
Port 2Yes
Slot 4
(full-wide compute nodes only)
Port 1Yes
Port 2Yes

Table 5 lists the Ethernet I/O adapters that are supported by the EN6131 40Gb Ethernet Switch.

Table 5. Supported Ethernet I/O adapters
DescriptionPart numberFeature code
40 Gb Ethernet
Flex System EN6132 2-port 40Gb Ethernet Adapter90Y3482A3HK
10 Gb Ethernet
Embedded 10Gb Virtual Fabric Adapter (2-port)None*None*
Flex System CN4022 2-port 10Gb Converged Adapter88Y5920A4K3
Flex System CN4052 2-port 10Gb Virtual Fabric Adapter00JY800A5RP
Flex System CN4054 10Gb Virtual Fabric Adapter (4-port)#90Y3554A1R1
Flex System CN4054R 10Gb Virtual Fabric Adapter (4-port)#00Y3306A4K2
Flex System EN4132 2-port 10Gb Ethernet Adapter90Y3466A1QY
Flex System EN4172 2-port 10Gb Ethernet Adapter00AG530A5RN
* The Embedded 10Gb Virtual Fabric Adapter is included in some models of the x240, x440, and X6 compute nodes.
# With 4-port 10 GbE adapters, only up to two adapter ports can be used with the EN6131 switches (one port per switch).

The adapters are installed in slots in each compute node. Figure 4 shows the locations of the slots in the x240 Compute Node. The positions of the adapters in the other supported servers are similar.

Location of the I/O adapter slots in the Flex System x240 Compute Node
Figure 4. Location of the I/O adapter slots in the Flex System x240 Compute Node

Connectors and LEDs

Figure 5 shows the front panel of the Flex System EN6131 40Gb Ethernet Switch.

Front panel of the IBM Flex System EN6131 40Gb Ethernet Switch
Figure 5. Front panel of the Flex System EN6131 40Gb Ethernet Switch

The front panel contains the following components:

  • LEDs that display the status of the module and the network:
    • Green power LED indicates that the module passed the power-on self-test (POST) with no critical faults and is operational.
    • Identify LED: This blue LED can be used to identify the module physically by illuminating it through the management software.
    • The fault LED (switch error) indicates that the module failed the POST or detected an operational fault.
  • Eighteen external QSFP+ ports for 10 Gbps, 20 Gbps, or 40 Gbps connections to the external network devices.
  • An Ethernet physical link LED and an Ethernet Tx/Rx LED for each external port on the module.
  • One mini-USB RS-232 console port that provides an additional means to configure the switch module. This mini-USB-style connector enables the connection of a special serial cable (the cable is optional, and it is not included with the switch. See the "Part number information" section for details).

Network cabling requirements

The network cables that can be used with the EN6131 40Gb Ethernet Switch are shown in Table 6.

Table 6. EN6131 40Gb Ethernet Switch network cabling requirements

TransceiverStandardCableConnector
40 Gb Ethernet
40GBASE-SR4 QSFP+ Transceiver (49Y7884)40GBASE-SR4Lenovo 10 m or 30 m MTP fiber optics cables (see Table 2); support for up to 100 m with OM3 multimode fiber or up to 150 m with OM4 multimode fiberMTP
Direct attach cable40GBASE-CR4Lenovo QSFP+ to QSFP+ DAC cables up to 7 m (see Table 2)QSFP+
Management ports
External 1 GbE management port1000BASE-TUTP Category 5, 5E, and 6 up to 100 metersRJ-45
External RS-232 management portRS-232DB-9-to-mini-USB or RJ-45-to-mini-USB console cable (comes with optional Management Serial Access Cable, 90Y9338)Mini-USB

Warranty

There is a 1-year, customer-replaceable unit (CRU) limited warranty. When installed in a chassis, these I/O modules assume your system’s base warranty and any Lenovo warranty service.

Physical specifications

The EN6131 40Gb Ethernet Switch has the following physical specifications:

Dimensions and weight (approximate):

  • Height: 30 mm (1.2 in.)
  • Width: 401 mm (15.8 in.)
  • Depth: 317 mm (12.5 in.)
  • Weight: 3.7 kg (8.1 lb)

Shipping dimensions and weight (approximate):
  • Height: 114 mm (4.5 in.)
  • Width: 508 mm (20.0 in.)
  • Depth: 432 mm (17.0 in.)
  • Weight: 4.1 kg (9.1 lb)

Agency approvals

The EN6131 40Gb Ethernet Switch conforms to the following standards:

  • Safety:
    • US/Canada: cULus
    • EU: IEC60950
    • International: CB
  • Environmental:
    • Type I / II
    • EU: IEC 60068-2-32: Fall Test

Typical configurations

A typical application of the end-to-end 40 Gb Ethernet connectivity includes high performance computing (HPC) interconnects and an enterprise data center's highly available production network.

HPC interconnect

In this configuration, there are two networks: a production network and an interconnect network. The production network consists of EN4093R 10Gb Scalable Switches that are connected to the RackSwitch™ G8264 10 GbE switches through redundant aggregated 10 Gb uplinks. The interconnect network consists of EN6131 switches connected to each other through 40 Gb aggregated uplinks. The topology is shown in Figure 6.

HPC interconnect network with the EN6131 40Gb Ethernet Switch
Figure 6. HPC interconnect network with the EN6131 40Gb Ethernet Switch

The solution components that are used in the HPC configuration that is shown in Figure 6 are listed in Table 7.

Table 7. Components that are used in the HPC interconnect with the EN6131 switch

Diagram
reference
DescriptionPart
number
Quantity
1RackSwitch G8264
2Flex System 10 Gb Virtual Fabric solution
Flex System Enterprise Chassis8721-A1xVaries
Flex System x240 Compute Node with Embedded 10Gb Virtual Fabric Adapter8737-x2xUp to 14 per chassis
Flex System Fabric EN4093R 10Gb Scalable Switch95Y33092 per chassis
Flex System 40 Gb HPC interconnect
Flex System EN6131 40Gb Ethernet Switch90Y93461 per chassis
Flex System EN6132 2-port 40Gb Ethernet Adapter90Y34821 per server

Note: You also need QSFP+ modules and optical cables or QSFP+ DACs (not shown in Table 6; see Table 2 for details) for the external connectivity.

40 Gb production network

In this configuration, a production network consists of EN6131 40Gb Ethernet Switches that are connected to the G8332 40 GbE upstream network switches through redundant aggregated 40 Gb uplinks (Figure 7).

Enterprise data center network with the EN6131 40Gb Ethernet Switch
Figure 7. Enterprise data center network with the EN6131 40Gb Ethernet Switch

The solution components that are used in the enterprise data center network configuration that is shown in Figure 7 are listed in Table 8.

Table 8. Components that are used in the enterprise data center network with the EN6131 switch
Diagram
reference
DescriptionPart
number
Quantity
1RackSwitch G8332
2Flex System 40 Gb solution
Flex System Enterprise Chassis8721-A1xVaries
Flex System x240 Compute Node8737-x1xUp to 14 per chassis
Flex System EN6131 40Gb Ethernet Switch90Y93462 per chassis
Flex System EN6132 2-port 40Gb Ethernet Adapter90Y34821 per server

Note: You also need QSFP+ modules and optical cables or QSFP+ DACs (not shown in Table 7; see Table 2 for details) for the external 40 Gb Ethernet connectivity.

Related publications

For more information, see the following EN6131 40Gb Ethernet Switch product publications, available from the Flex System Information Center at the following website:
http://publib.boulder.ibm.com/infocenter/flexsys/information/index.jsp

  • Installation and User's Guide

Here are other useful references:

Special Notices

This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment.

Profile

Publish Date
06 August 2013

Last Update
20 July 2015


Rating: Not yet rated


Author(s)

IBM Form Number
TIPS0911