IBM Flex System EN4023 10Gb Scalable Switch

IBM Redbooks Product Guide

Abstract

The IBM® Flex System™ EN4023 10Gb Scalable Switch is a high-performance 10 Gigabit Ethernet (GbE) embedded switch that supports the most demanding business applications. Clients deploying the new Brocade Virtual Cluster Switching (VCS) Fabric can now extend the features to the Flex System chassis using the EN4023 switch module. It is specifically designed to improve network utilization, maximize application availability, increase scalability, and dramatically simplify network architecture in virtualized data centers, as described in this IBM Redbooks® Product Guide.

Changes in the November 22 update:
* Added support for SFP+ direct-attach cables

Contents


Table of contents


Introduction

The IBM® Flex System EN4023 10Gb Scalable Switch is a high-performance 10 Gigabit Ethernet (GbE) embedded switch that supports the most demanding business applications. Clients deploying the new Brocade Virtual Cluster Switching (VCS) Fabric can now extend the features to the Flex System chassis using the EN4023 switch module. It is specifically designed to improve network utilization, maximize application availability, increase scalability, and dramatically simplify network architecture in virtualized data centers, as described in this IBM Redbooks® Product Guide.

The following figure shows the EN4023 10Gb Scalable Switch.

IBM Flex System EN4023 10Gb Scalable Switch
Figure 1. IBM Flex System EN4023 10Gb Scalable Switch

Did you know?

The EN4023 switches feature Brocade VCS Fabric technology that enables organizations to build high-performance, cloud-optimized data centers while preserving existing network designs and cabling, and gaining active-active server connections. For scale-out fabric architectures, Brocade VCS Fabric technology allows organizations to flatten network designs, provide virtual machine (VM) mobility without network reconfiguration, and manage the entire fabric more efficiently.

With Dynamic Ports on Demand (DPOD), ports are licensed as they come online; therefore, the EN4023 allows you to buy only the ports that you need, when you need them. The base module includes 24 port licenses for 10 GbE connectivity that can be applied to the internal and external ports. You then have the flexibility of turning on more 10 GbE ports and 40 GbE uplinks when you need them by using IBM Features on Demand (FoD) licensing capabilities that provide "pay as you grow" scalability.

Part number information

The EN4023 module is initially licensed for 24 ports (internal or external 10 GbE connectivity). Further ports can be activated, including 16 additional 10 GbE ports and two 40 Gb external uplink ports with the FoD Upgrade 1 license option, and 16 more 10 GbE ports with the FoD Upgrade 2 license option. Upgrade 1 and Upgrade 2 can be applied independently of each other. The following table shows the part numbers for ordering the switches and the upgrades.

Table 1. Part numbers and feature codes for ordering

DescriptionPart numberFeature code
(x-config / e-config)
Switch module
IBM Flex System EN4023 10Gb Scalable Switch94Y5212ESWD / ESWD
Features on Demand upgrades
IBM Flex System EN4023 10Gb Scalable Switch (FoD 1)94Y5158ESWE / ESWE
IBM Flex System EN4023 10Gb Scalable Switch (FoD 2)94Y5159ESWF / ESWF

The part numbers for the switches include the following items:
  • One IBM Flex System EN4023 10Gb Scalable Switch
  • Important Notices Flyer
  • Warranty Flyer
  • Documentation CD-ROM

Note: Small form factor (SFP) and small form-factor pluggable plus (SFP+) transceivers are not included with the switch. They must be ordered separately (see Table 2).

The part numbers for the Features on Demand upgrades include the following items:
  • Features on Demand Activation Flyer
  • Upgrade authorization letter

The switch does not include a serial management cable; however, IBM Flex System Management Serial Access Cable, 90Y9338, is supported. It contains two cables, a mini-USB-to-RJ45 serial cable and a mini-USB-to-DB9 serial cable, either of which can be used to connect to the switch locally for configuration tasks.

The base switch and upgrades are specified:
  • 94Y5212 is the part number for the physical device, and it comes with 24 ports enabled (any combination of internal and external 10 GbE ports, except 40 GbE uplinks). All external 10 Gb ports are connections that are based on SFP+.
  • 94Y5158 (FoD Upgrade 1) can be applied on the base switch or on top of the FoD Upgrade 2 to enable 16 additional 10 GbE ports (internal and external). The upgrade also enables two 40 Gb uplinks with quad small form factor pluggable plus (QSFP+) connectors.
  • 94Y5159 (FoD Upgrade 2) can be applied on the base switch or on top of the FoD Upgrade 1 to enable 16 additional 10 GbE ports (internal and external).

The following table lists supported port combinations on the switch and required upgrades.

Table 2. Supported port combinations
Supported port combinations
Quantity required
Base switch, 94Y5212Upgrade 1, 94Y5158Upgrade 2, 94Y5159
  • 24x 10 GbE ports (internal and external)
1
0
0
  • 40x 10 GbE ports (internal and external)
1
0
1
  • 40x 10 GbE ports (internal and external)
  • 2x external 40 GbE QSFP+ ports
1
1
0
  • 56x 10 GbE ports (internal and external)
  • 2x external 40 GbE QSFP+ ports
1
1
1

Supported cables and transceivers

The following table lists the supported transceivers and cables.

Table 3. Supported transceivers and cables.

DescriptionPart numberFeature code
(x-config / e-config)
Serial console cables
IBM Flex System Management Serial Access Cable Kit90Y9338A2RR / None
SFP+ transceivers - 10 GbE
IBM SFP+ SR Transceiver46C34475053 / EB28
IBM SFP+ LR Transceiver00D6180A3NZ / ECB9
Brocade VDX SFP+ LR Transceiver95Y0540A3AB / EB37
Brocade 10Gb SFP+ SR Optical Transceiver49Y42160069 / EB3C
SFP+ direct-attach cables - 10 GbE
1m 10GE Twinax Act Copper SFP+81Y8295A18M / EN01
3m 10GE Twinax Act Copper SFP+81Y8296A18N / EN02
5m 10GE Twinax Act Copper SFP+81Y8297A18P / EN03
1m IBM Active DAC SFP+ Cable95Y0323A25A / None
3m IBM Active DAC SFP+ Cable95Y0326A25B / None
5m IBM Active DAC SFP+ Cable95Y0329A25C / None
QSFP+ transceiver and cables - 40 GbE
IBM QSFP+ 40GBASE-SR Transceiver
(Requires either cable 90Y3519 or cable 90Y3521)
49Y7884A1DR / EB27
10m IBM MTP Fiber Optical Cable (requires transceiver 49Y7884)90Y3519A1MM / EB2J
30m IBM MTP Fiber Optical Cable (requires transceiver 49Y7884)90Y3521A1MN / EC2K

Benefits

The EN4023 10Gb Scalable Switch is designed for easy integration into a Brocade Virtual Cluster Switching (VCS) environment. Brocade VCS Fabric technology simplifies network design and operations for a more automated and efficient network. The switch offers the following benefits:

  • High performance and low latency:
    • With 10 Gb connections, the Flex System EN4023 switch can help your organization reduce network congestion, improve application performance, and meet the capacity required by 10 Gb servers. The 40 Gb uplinks can easily aggregate high-bandwidth traffic and reduce bottlenecks, helping your network operate at its peak performance.
    • Provides ultra-low any-to-any port latency of 850 nanoseconds.
  • Simplicity and automation:
    • Elastic, self-forming, and self-healing fabrics: Allows new switches added to the fabric to inherit the fabric configuration, providing a highly resilient, operationally scalable foundation.
    • Auto Fabric Provisioning: Automatically downloads the latest software image from the server, reducing configuration time.
  • Efficiency and resiliency:
    • Optimized east-west traffic: Embedded switching moves traffic through any active path and avoids the multiple hops found in tiered tree topologies.
    • Efficient multipathing across Layers 1 - 3: Delivers efficiently load-balanced multipathing at Layers 1 - 3, with multiple Layer 3 gateways for more effective link utilization.
  • Optimized for virtualization:
    • Brocade VM-Aware Network Automation: Eliminates the manual configuration of port profiles when a VM is added to the fabric or moved.
    • Automatic Migration of Port Profiles (AMPP): Applies VM and networking policies to a VM as it moves within the Brocade VCS Fabric.
    • Proactive Monitoring.
    • Brocade Fabric Watch monitors the health of certain switch components and, based on the threshold set, declares each component as marginal or down.

Features and specifications

The IBM Flex System EN4023 10Gb Scalable Switch has the following features and specifications:

  • Internal ports:
    • Forty-two internal full-duplex 10 Gigabit ports.
    • Two internal full-duplex 1 GbE ports that are connected to the chassis management module.
  • External ports:
    • Fourteen ports for 10 Gb Ethernet SFP+ transceivers (support for 10GBASE-SR or 10GBASE-LR). SFP+ modules are not included and must be purchased separately.
    • Two ports for 40 Gb Ethernet QSFP+ transceivers (Ports are disabled, by default. An optional FoD license is required to activate them.) QSFP+ modules are not included and must be purchased separately.
    • One RS-232 serial port (mini-USB connector) that provides an additional means to configure the switch module.
  • Converged Enhanced Ethernet (CEE) features:
    • Priority-based Flow Control (PFC): IEEE 802.1Qbb
    • Enhanced Transmission Selection (ETS): IEEE 802.1Qaz
    • Data Center Bridging eXchange (DCBX)
  • Layer 2 features:
    • Layer 2 Virtual Local Area Networks (VLANs): 4096 VLANs
    • VLAN encapsulation 802.1Q
    • Spanning Tree Protocol (STP) IEEE 802.1D
    • Rapid Spanning Tree Protocol (RSTP)
    • Multiple Spanning Tree (MSTP) (802.1s): 32 instances
    • Per-VLAN Spanning Tree (PVST+/PVRST+)
    • STP PortFast and PortFast BDPU Guard
    • STP Root Guard
    • Link Aggregation Control Protocol (LACP) IEEE 802.3ad
    • Internet Group Management Protocol (IGMP) v1/v2 snooping
    • Pause frames (802.3x)
  • Layer 2 security:
    • Ingress Layer 2 Media Access Control (MAC) Access Control Lists (ACLs)
    • Standard and extended ACLs
    • VLAN-based ACLs (VACLs)
    • Port-based ACLs (PACLs)
    • ACL statistics
    • Port-based Network Access Control: IEEE 802.1x
  • Layer 2 Quality of Service (QoS):
    • Eight priority levels for QoS
    • IEEE 802.1p Class of Service (CoS)
    • Per-port QoS configuration
    • Eight queues per port
    • CoS trust: IEEE 802.1p
    • Scheduling: Strict Priority (SP), Shaped Deficit Weighted Round-Robin (SDWRR)
  • Layer 3 features:
    • Open Shortest Path First (OSPF)
    • Border Gateway Protocol (BGP)
    • Virtual Router Redundancy Protocol (VRRP) and VRRP-E support
    • Static routes
  • Brocade VCS Fabric technology features:
    • Automatic Fabric Formation
    • Distributed Fabric Services
    • Transparent LAN Services
    • Virtual Link Aggregation Group (vLAG) spanning multiple physical switches
    • Switch Beaconing
    • Distributed Configuration Management
    • Transparent Interconnection of Lots of Links (TRILL)
    • Equal Cost Multi-Path (ECMP)
    • Automatic Migration of Port Profiles (AMPP)
    • VM-Aware Network Automation
  • Management security:
    • RADIUS
    • TACACS+
    • Secure Shell (SSHv2)
    • Management access ACLs

The following software feature comes with the switch modules:
  • Brocade Web Tools

The switch supports the following fabric management:
  • Web interface through Web Tools
  • Command-line interface (CLI) through Telnet or Secure Shell V2 (SSHv2)
  • A terminal emulation program connection to the serial port interface
  • Brocade Network Advisor
  • Switch's Simple Network Management Protocol (SNMP) agent

Supported standards

The switches support the following standards:

  • IEEE 802.1AB Data Center Bridging Capability Exchange Protocol (DCBX)
  • IEEE 802.1p Class of Service (CoS) prioritization
  • IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled)
  • IEEE 802.1Qbb Priority-Based Flow Control (PFC)
  • IEEE 802.1Qaz Enhanced Transmission Selection (ETS)
  • IEEE 802.3 Ethernet
  • IEEE 802.3ab 1000BASE-T copper twisted pair Gigabit Ethernet
  • IEEE 802.3ad Link Aggregation Control Protocol
  • IEEE 802.3ae 10GBASE-SR short range fiber optics 10 Gb Ethernet
  • IEEE 802.3ae 10GBASE-LR long range fiber optics 10 Gb Ethernet
  • IEEE 802.3ap 10GBASE-KR backplane 10 Gb Ethernet
  • IEEE 802.3ba 40GBASE-SR4 short range fiber optics 40 Gb Ethernet
  • IEEE 802.3u 100BASE-TX Fast Ethernet
  • IEEE 802.3x Full-duplex Flow Control
  • SFF-8431 10GSFP+Cu SFP+ Direct Attach Cable

Supported chassis and adapters

The I/O modules are installed in switch bays in the rear of the IBM Flex System Enterprise Chassis, as shown in the following figure. I/O modules are normally installed in pairs because ports on the I/O adapters that are installed in the compute nodes are routed to two switch bays for redundancy and performance.

Location of the switch bays in the IBM Flex System Enterprise Chassis
Figure 2. Location of the switch bays in the IBM Flex System Enterprise Chassis

The connections between the adapters that are installed in the compute nodes to the EN4023 installed in the I/O bays in the chassis are shown in the following figure. The figure shows both half-wide servers, such as the x240 with two adapters, and full-wide servers, such as the x440 with four adapters.

Logical layout of the interconnects between I/O adapters and the EN4023 modules
Figure 3. Logical layout of the interconnects between I/O adapters and the EN4023 modules

The EN4023 10Gb Scalable Switches can be installed in bays 1, 2, 3, and 4 of the Enterprise Chassis. A supported adapter must be installed in the corresponding slot of the compute node (slot A1 when interconnect modules are installed in bays 1 and 2 or slot A2 when the modules are in bays 3 and 4). With four-port or eight-port adapters, an optional FoD Upgrade might be required for the switch to allow communications on all ports (However, with eight-port adapters, only six adapter ports can be used, and the two remaining ports are not used.). (See Table 2 for details.)

In compute nodes that have an integrated dual-port 10 GbE network interface controller (NIC), NIC ports are routed to bays 1 and 2 with a specialized periscope connector, and the adapter in slot A1 is not required. However, when needed, the periscope connector can be replaced with the adapter. In such a case, the integrated NIC is disabled.

The following table shows the connections between adapters that are installed in the compute nodes to the EN4023 modules installed in the I/O bays in the chassis.

Table 4. Adapter to I/O bay correspondence

I/O adapter slot
in the server
Port on the adapterCorresponding I/O bay in the chassis
Bay 1Bay 2Bay 3Bay 4
x222 integrated LOMPort 1Yes
Port 2Yes
Port 1Yes
Port 2Yes
Slot 1 (or integrated dual-port LOM - ports 1 and 2)Port 1Yes
Port 2Yes
Port 3*Yes
Port 4*Yes
Port 5*Yes
Port 6*Yes
Slot 2Port 1Yes
Port 2Yes
Port 3*Yes
Port 4*Yes
Port 5*Yes
Port 6*Yes
Slot 3
(full-wide compute nodes only)
Port 1Yes
Port 2Yes
Port 3*Yes
Port 4*Yes
Port 5*Yes
Port 6*Yes
Slot 4
(full-wide compute nodes only)
Port 1Yes
Port 2Yes
Port 3*Yes
Port 4*Yes
Port 5*Yes
Port 6*Yes
* Depending on the number of compute nodes and uplinks, Ports 3, 4, 5, and 6 might require an optional FoD port upgrade of the EN4023 (see Table 2).

The following table lists the I/O adapters that are supported by the EN4023 10Gb Scalable Switches.

Table 5. Supported network adapters
DescriptionPart
number
Feature code
(x-config / e-config)
Supported by
EN4023
40 Gb Ethernet
IBM Flex System EN6132 2-port 40Gb Ethernet Adapter90Y3482A3HK / A3HKNo
10 Gb Ethernet
Embedded 10Gb Virtual Fabric Adapter (x222; dual 2-port)NoneNone / NoneYes
Embedded 10Gb Virtual Fabric Adapter (x240, x440; 2-port)*NoneNone / NoneYes
IBM Flex System CN4022 2-port 10Gb Converged Adapter88Y5920A4K3 / A4K3Yes
IBM Flex System CN4054 10Gb Virtual Fabric Adapter (4-port)90Y3554A1R1 / NoneYes
IBM Flex System CN4054R 10Gb Virtual Fabric Adapter (4-port)00Y3306A4K2 / A4K2Yes
IBM Flex System CN4058 8-port 10Gb Converged AdapterNoneNone / EC24Yes#
IBM Flex System EN4054 4-port 10Gb Ethernet AdapterNoneNone / 1762Yes
IBM Flex System EN4132 2-port 10Gb Ethernet Adapter90Y3466A1QY / EC2DYes
IBM Flex System EN4132 2-port 10Gb RoCE AdapterNoneNone / EC26Yes
1 Gb Ethernet
Embedded 1 Gb Ethernet controller (x220; 2-port)NoneNone / NoneYes
IBM Flex System EN2024 4-port 1Gb Ethernet Adapter 49Y7900A10Y / 1763Yes
* The Embedded 10Gb Virtual Fabric Adapter is included in some models of the x240 and x440.
# With eight-port adapters, only six adapter ports can be used with the EN4023, and the remaining two ports are not used.

The adapters are installed in slots in each compute node. The following figure shows the locations of the slots in the x240 Compute Node. The positions of the adapters in the other supported servers are similar.

 Location of the I/O adapter slots in the IBM Flex System x240 Compute Node
Figure 4. Location of the I/O adapter slots in the IBM Flex System x240 Compute Node

Connectors and LEDs

The following figure shows the front panel of the EN4023 10Gb Scalable Switch.

Front panel of the IBM Flex System EN4023 10Gb Scalable Switch
Figure 5. Front panel of the IBM Flex System EN4023 10Gb Scalable Switch

The front panel contains the following components:

  • Light-emitting diodes (LEDs) that display the status of the interconnect module and the network:
    • The OK LED indicates that the interconnect module passed the power-on self-test (POST) with no critical faults and is operational.
    • Identify: This blue LED can be used to identify the module physically by illuminating it through the management software.
    • The error LED (switch module error) indicates that the module failed the POST or detected an operational fault.
  • One mini-USB RS-232 console port that provides an additional means to configure the interconnect module. This mini-USB-style connector enables the connection of a special serial cable. (The cable is optional, and it is not included with the interconnect module. For more information, see the "Part number information" section.)
  • One 1 Gb Ethernet RJ-45 port for switch configuration and management.
  • Fourteen external SFP+ ports for 10 Gb connections to external Ethernet devices.
  • Two external QSFP+ port connectors to attach QSFP+ modules for a single 40 Gb uplink per port.
  • An Ethernet link OK LED and an Ethernet Tx/Rx LED for each external port on the interconnect module.

Network cabling requirements

The network cables that can be used with the EN4023 are shown in the following table.

Table 6. EN4023 network cabling requirements

TransceiverStandardCableConnector
40 Gb Ethernet
IBM QSFP+ 40GBASE-SR Transceiver (49Y7884)40GBASE-SR410 m or 30 m IBM MTP fiber optics cables (see Table 3)MTP
10 Gb Ethernet
SFP+ SR Transceiver (46C3447, 49Y4216)10GBASE-SR850 nm multimode fiber cable (50 µ or 62.5 µ) up to 300 mLC
SFP+ LR Transceiver (90Y9412, 95Y0540)10GBASE-LR1310 nm single-mode fiber cable up to 10 kmLC
Direct attach cable10GSFP+CuUp to 5 m SFP+ copper DACs (see Table 3)SFP+
Management ports
External 1 GbE management port1000BASE-TUTP Category 5, 5E, and 6 up to 100 metersRJ-45
External RS-232 management portRS-232DB-9-to-mini-USB or RJ-45-to-mini-USB console cable (comes with optional Management Serial Access Cable, 90Y9338)Mini-USB

Warranty

The IBM Flex System EN4023 carries a 1-year, customer-replaceable unit (CRU) limited warranty. When installed in a chassis, these I/O modules assume your system’s base warranty and any IBM ServicePac® upgrade.

Physical specifications

Dimensions and weight of the EN4023 (approximate):

  • Height: 30 mm (1.2 in.)
  • Width: 402 mm (15.8 in.)
  • Depth: 297 mm (11.7 in.)
  • Weight: 3.6 kg (7.9 lb)

Shipping dimensions and weight (approximate):
  • Height: 114 mm (4.5 in.)
  • Width: 508 mm (20.0 in.)
  • Depth: 432 mm (17.0 in.)
  • Weight: 4.1 kg (9.1 lb)

Agency approvals

The EN4023 conforms to the following regulations:

  • UL 60950
  • CAN/CSA-C22.2 No 60950
  • IEC/EN 60950
  • FCC 47 CFR Part 15, Class A
  • AS/NZS CISPR 22, Class A
  • CISPR 22, Class A
  • CE Mark (EN55022 Class A, EN55024, EN61000-3-2, EN61000-3-3)
  • ICES003, Class A
  • VCCI, Class A
  • KN22, Class A

Typical configurations

The connectivity topology, which can be used with the EN4023 switches and the Brocade MLX upstream network devices, is shown in the following figure.

EN4023 connectivity topology - Virtual Link Aggregation
Figure 6. EN4023 connectivity topology - Virtual Link Aggregation

In this access-layer topology, aggregation is split between two physical switches, and each EN4023 is connected to both ToR switches through static or Link Aggregation Control Protocol (LACP)-aggregated links. Compute Node's NICs are configured in an active/active NIC teaming. This topology can be used to insert Brocade VCS Fabric technology into existing infrastructure, because it fully interoperates with existing LAN protocols, services, and architectures.

The connectivity topology, which can be used with the EN4023 switches and the Brocade VDX upstream network devices, is shown in the following figure.

EN4023 connectivity topology - Inter-Switch Links
Figure 7. EN4023 connectivity topology - Inter-Switch Links

In this topology, EN4023 switches are connected to the Brocade VDX switches via Inter-Switch Links (ISLs) forming a scale-out self-aggregated data center edge fabric that is based on Brocade VCS technology. Such a fabric, where STP is eliminated, is highly resilient and efficient, and it allows network administrators to manage the entire network as a single logical switch.

With both topologies, each port of the compute node's NIC can be divided into up to four virtual NICs (vNICs), adding more granularity in configuring virtualized network environments.

EN4023 in the VCS network

The following figure shows a sample scenario where the EN4023 is used as a part of the data center edge network built with VCS technology.

EN4023 in the VCS network with the Brocade VDX ToR
Figure 8. EN4023 in the VCS network with the Brocade VDX ToR

The following table refers to the corresponding numbers in Figure 8.

Table 7. EN4023 in the VCS network with the Brocade VDX ToR (Figure 8)

Diagram
reference
DescriptionPart
number
Quantity
1IBM Flex System VCS solution
IBM Flex System x240 Compute Node with Embedded Virtual Fabric AdapterVariesVaries
IBM Flex System EN4023 10Gb Scalable Switch94Y5212Two per chassis
2Brocade VDX Switch

Note: You also need SFP+ modules and optical cables (not shown in Table 7; see Table 3 for details) for the external 10 Gb Ethernet connectivity.

Related publications

For more information, see the following EN4023 10Gb Scalable Switch product publication, which is available from the IBM Flex System Information Center at
http://publib.boulder.ibm.com/infocenter/flexsys/information/index.jsp:

  • Installation and User Guide

The following documents are other useful references:



Special Notices

This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment.

Profile

Publish Date
08 October 2013

Last Update
22 November 2013


Rating: Not yet rated


Author(s)

IBM Form Number
TIPS1070