IBM Flex System CN4054 10Gb Virtual Fabric Adapter and EN4054 4-port 10Gb Ethernet Adapter

The IBM Flex System™ CN4054 10Gb Virtual Fabric Adapter is a 4-port 10Gb converged network adapter (CNA) for Intel processor-based compute nodes that can scale to up to 16 virtual ports and that supports Ethernet, iSCSI, and FCoE. The adapter supports up to eight virtual NIC (vNIC) devices, where each physical 10 GbE port can be divided into four virtual ports with flexible bandwidth allocation. The CN4054 Virtual Fabric Adapter Upgrade adds FCoE and iSCSI hardware initiator functionality.

The IBM Flex System EN4054 4-port 10Gb Ethernet Adapter is based on similar hardware but simply operates as a straight 4-port 10 GbE adapter. The EN4054 it is supported by POWER7 processor-based compute nodes. Figure 1 shows the adapters (they have the same physical appearance).


Figure 1. IBM Flex System CN4054 10Gb Virtual Fabric Adapter and EN4054 4-port 10Gb Ethernet Adapter


Did you know?

This CN4054 is based on industry-standard PCIe architecture and offers the flexibility to operate as a Virtual NIC Fabric Adapter or as a quad-port 10Gb or 1Gb Ethernet device. Because this adapter supports up to 16 virtual NICs on a single quad-port Ethernet adapter, you see benefits in cost, power/cooling, and data center footprint by deploying less hardware. Compute nodes like the x240 support up to two of these adapters for a total of 32 virtual NICs per system.


Part number information

IBM Flex System is a new category of computing that integrates multiple server architectures, networking, storage, and system management capability into a single system that is easy to deploy and manage. IBM Flex System has full built-in virtualization support of servers, storage, and networking to speed provisioning and increased resiliency. In addition, it supports open industry standards, such as operating systems, networking and storage fabrics, virtualization, and system management protocols, to easily fit within existing and future data center environments. IBM Flex System is scalable and extendable with multi-generation upgrades to protect and maximize IT investments.

Table 1. Ordering part numbers and feature codes

Description	Part number	Feature code (x-config)	Feature code (e-config)
IBM Flex System CN4054 10Gb Virtual Fabric Adapter	90Y3554	A1R1	1759
IBM Flex System CN4054 Virtual Fabric Adapter Upgrade*	90Y3558	A1R0	1760
IBM Flex System EN4054 4-port 10Gb Ethernet Adapter	None	None	1762

* One upgrade needed per adapter

The CN4054 10Gb Virtual Fabric Adapter part number includes the following items:

• One adapter
• Documentation CD
• Important Notices flyer

The IBM Flex System CN4054 10Gb Virtual Fabric Adapter can be upgraded to run storage protocols iSCSI and FCoE by applying the upgrade licence, IBM Flex System CN4054 Virtual Fabric Adapter Upgrade, part number 90Y3558. The IBM Flex System EN4054 4-port 10Gb Ethernet Adapter does not support this upgrade.


Features

Both the IBM Flex System CN4054 10Gb Virtual Fabric Adapter and the EN4054 4-port 10Gb Ethernet Adapter have these features:

• Four-port 10 Gb Ethernet adapter
• Dual-ASIC Emulex BladeEngine 3 (BE3) controller
• Connection to either 1 Gb or 10 Gb data center infrastructure (1 Gb and 10 Gb auto-negotiation)
• PCI Express 3.0 x8 host interface (The p260 and p460 support PCI Express 2.0 x8.)
• Full-duplex (FDX) capability
• Bus-mastering support
• Direct memory access (DMA) support
• Preboot Execution Environment (PXE) support
• IPv4/IPv6 TCP, UDP checksum offload
  • Large send offload (LSO)
  • Large receive offload
  • Receive side scaling (RSS)
  • IPv4 TCP Chimney Offload
  • TCP Segmentation Offload
• VLAN insertion and extraction
• Jumbo frames up to 9000 bytes
• Load balancing and failover support, including adapter fault tolerance (AFT), switch fault tolerance (SFT), adaptive load balancing (ALB), teaming support, and IEEE 802.3ad
• Enhanced Ethernet (draft)
  • Enhanced Transmission Selection (ETS) (P802.1Qaz)
  • Priority-based Flow Control (PFC) (P802.1Qbb)
  • Data Center Bridging Capabilities eXchange Protocol, CIN-DCBX and CEE-DCBX (P802.1Qaz)
• Supports Serial over LAN (SoL)
• Total Max Power: 23.1 W

The CN4054 10Gb Virtual Fabric Adapter also supports the following additional features:

• Operates either as a 4-port 1/10 Gb Ethernet adapter or supports up to 16 vNICs.
• Supports Unversal Fabric Port (UFP)
• In virtual NIC (vNIC) mode, it supports:
  • Virtual port bandwidth allocation in 100 Mbps increments.
  • Up to 16 virtual ports per adapter (four per port).
  • With the CN4054 Virtual Fabric Adapter Upgrade, 90Y3558, four of the 16 vNICs (one per port) support iSCSI or FCoE.
• Supports for two vNIC modes: IBM® Virtual Fabric Mode and Switch Independent Mode
• Wake On LAN support.
• With the CN4054 Virtual Fabric Adapter Upgrade, 90Y3558, the adapter adds FCoE and iSCSI hardware initiator support.
  • iSCSI support is implemented as a full offload and presents an iSCSI adapter to the operating system.
• TCP Offload Engine (TOE) support with Windows Server 2003, 2008, and 2008 R2 (TCP Chimney) and Linux.
  • Connection and its state are passed to the TCP offload engine.
  • Data transmit and receive is handled by adapter.
  • Supported with iSCSI.

CN4054 modes of operation

The CN4054 10Gb Virtual Fabric Adapter supports two vNIC modes of operation in addition to pNIC mode:

• IBM Virtual Fabric Mode. This mode only works in conjunction with IBM Flex System Fabric EN4093 10Gb Scalable Switch installed in the chassis. In this mode, the adapter communicates with the switch module to obtain vNIC parameters (using DCBX). Also, a special tag within each data packet is added and later removed by the NIC and switch for each vNIC group to maintain separation of the virtual channels.

  In vNIC mode, each physical port is divided into four virtual ports, providing a total of 16 virtual NICs per adapter. The default bandwidth for each vNIC is 2.5 Gbps. Bandwidth for each vNIC can be configured at the EN4093 switch from 100 Mbps to 10 Gbps, up to a total of 10 Gb per physical port. The vNICs can also be configured to have 0 bandwidth if you must allocate the available bandwidth to fewer than eight vNICs. In IBM Virtual Fabric Mode, you can change the bandwidth allocations through the EN4093 switch user interfaces without requiring a reboot of the server.

  When storage protocols are enabled on the adapter (using CN4054 Virtual Fabric Adapter Upgrade, 90Y3558), six ports are Ethernet, and two ports are either iSCSI or FCoE

• Switch Independent Mode, where the adapter works with the following switches:
  • IBM Flex System Fabric EN4093 10Gb Scalable Switch
  • IBM Flex System EN4091 10Gb Ethernet Pass-thru and a top-of-rack (TOR) switch
  
  Switch Independent Mode offers the same capabilities as IBM Virtual Fabric Mode in terms of the number of vNICs and the bandwidth that each can be configured to have. However, Switch Independent Mode extends the existing customer VLANs to the virtual NIC interfaces. The IEEE 802.1Q VLAN tag is essential to the separation of the vNIC groups by the NIC adapter or driver and the switch. The VLAN tags are added to the packet by the applications or drivers at each end station rather than by the switch.
• In pNIC mode, the expansion card can operate as a standard 10 Gbps or 1 Gbps 4-port Ethernet expansion card. When in pNIC mode, the expansion card functions with any of the following I/O modules:
  • IBM Flex System Fabric EN4093 10Gb Scalable Switch
  • IBM Flex System EN4091 10Gb Ethernet Pass-thru and a top-of-rack (TOR) switch
  • IBM Flex System EN2092 1Gb Ethernet Scalable Switch
  
  In pNIC mode, the adapter with the CN4054 Virtual Fabric Adapter Upgrade, 90Y3558, applied operates in traditional Converged Network Adapter (CNA) mode with four ports of Ethernet and four ports of storage (iSCSI or FCoE) available to the operating system.

Supported servers

The following table lists the IBM Flex System compute nodes that support the adapters.

Table 2. Supported servers and maximum quantities

Description	Part number	x220	x240	x440	p24L	p260	p460
IBM Flex System CN4054 10Gb Virtual Fabric Adapter	90Y3554	Yes	Yes	Yes	No	No	No
IBM Flex System CN4054 Virtual Fabric Adapter Upgrade	90Y3558	Yes	Yes	Yes	No	No	No
IBM Flex System EN4054 4-port 10Gb Ethernet Adapter	Feature 1762	No	No	No	Yes	Yes	Yes

See IBM ServerProven at the following web address for the latest information about the expansion cards that are supported by each blade server type:
http://ibm.com/servers/eserver/serverproven/compat/us/

I/O adapter cards are installed in the slot in supported servers, such as the x240, as highlighted in the following figure.


Figure 2. Location of the I/O adapter slots in the IBM Flex System x240 Compute Node


Embedded 10Gb Virtual Fabric Adapter

Some models of the x240 (those with a model number of the form x2x) include an Embedded 10Gb Virtual Fabric Adapter (VFA, also known as LAN on Motherboard or LOM) built into the system board. Each x240 model that includes the embedded 10Gb VFA also has the Compute Node Fabric Connector installed in I/O connector 1 (and physically screwed onto the system board) to provide connectivity to the Enterprise Chassis midplane.

The Fabric Connector enables port 1 on the Embedded 10Gb VFA to be routed to I/O module bay 1 and port 2 to be routed to I/O module bay 2. The Fabric Connector can be unscrewed and removed, if required, to allow the installation of an I/O adapter on I/O connector 1.

The Embedded 10Gb VFA is based on the same Emulex BladeEngine 3 (BE3) controller and offers two 10 GbE ports. The 2-port Embedded 10Gb VFA offers the same features as the 4-port CN4054 10Gb Virtual Fabric Adapter. The Embedded 10Gb VFA also supports iSCSI and FCoE using the upgrade license listed in the following table.

Table 3. Feature on Demand upgrade for FCoE and iSCSI support

Part number	Description
90Y9310	IBM Flex System Embedded 10Gb Virtual Fabric Upgrade

Supported I/O modules

These adapters can be installed in any I/O adapter slot of a supported IBM Flex System compute node. One or two compatible 1 Gb or 10 Gb I/O modules must be installed in the corresponding I/O bays in the chassis. The following table lists the switches that are supported. When connected to the 1Gb switch, the adapter will operate at 1 Gb speeds.

Table 4. I/O modules and upgrades for use with the CN4054 and EN4054 adapters

Description	Part number	Feature code (x-config)	Feature code (e-config)
IBM Flex System Fabric EN4093 10Gb Scalable Switch	49Y4270	A0TB	3593
IBM Flex System Fabric EN4093 10Gb Scalable Switch (Upgrade 1)*	49Y4798	A1EL	3596
IBM Flex System EN4091 10Gb Ethernet Pass-thru**	88Y6043	A1QV	3700
IBM Flex System EN2092 1Gb Ethernet Scalable Switch	49Y4294	A0TF	3598
IBM Flex System EN2092 1Gb Ethernet Scalable Switch (Upgrade 1)*	90Y3562	A1QW	3594

* Upgrade 1 is required for the EN4093 and EN2092 to use all four ports on the adapter
** The EN4091 Pass-thru has only 14 internal ports and so only supports ports 1 and 2 of the adapter

Installing two switches means that all four ports of the adapter are enabled. However, the use of all four ports also requires Upgrade 1 of the supported 10 GbE and 1 GbE switches be applied. Upgrade 1, as listed in the preceding table, enables 28 internal ports on each switch. The EN4091 Pass-thru has only 14 internal ports and so only supports ports 1 and 2 of the adapter.

The following table shows the connections between adapters installed in the compute nodes to the switch bays in the chassis.

Table 5. Adapter to I/O bay correspondence

I/O adapter slot in the server	Port on the adapter	Corresponding I/O module bay in the chassis
Slot 1	Port 1	Module bay 1
	Port 2	Module bay 2
	Port 3*	Module bay 1
	Port 4*	Module bay 2
Slot 2	Port 1	Module bay 3
	Port 2	Module bay 4
	Port 3*	Module bay 3
	Port 4*	Module bay 4
Slot 3 (full-wide compute nodes only)	Port 1	Module bay 1
	Port 2	Module bay 2
	Port 3*	Module bay 1
	Port 4*	Module bay 2
Slot 4 (full-wide compute nodes only)	Port 1	Module bay 3
	Port 2	Module bay 4
	Port 3*	Module bay 3
	Port 4*	Module bay 4

* Ports 3 and 4 require Upgrade 1 of either the EN4093 10Gb or the EN2092 1Gb switch. The EN4091 Pass-thru only supports ports 1 and 2 (and only when two I/O modules are installed).

The connections between the adapters installed in the compute nodes to the switch bays in the chassis are shown diagrammatically in the following figure. The figure shows both half-wide servers, such as the x240 with two adapters, and full-wide servers, such as the p460 with four adapters.


Figure 3. Logical layout of the interconnects between I/O adapters and I/O modules


Operating system support

The IBM Flex System CN4054 10Gb Virtual Fabric Adapter supports the following operating systems:

• Microsoft Windows Server 2008 R2
• Microsoft Windows Server 2008, Datacenter x64 Edition
• Microsoft Windows Server 2008, Enterprise x64 Edition
• Microsoft Windows Server 2008 HPC Edition
• Microsoft Windows Server 2008, Standard x64 Edition
• Microsoft Windows Server 2008, Web x64 Edition
• Red Hat Enterprise Linux 5 Server x64 Edition
• Red Hat Enterprise Linux 5 Server with Xen x64 Edition
• Red Hat Enterprise Linux 6 Server x64 Edition
• SUSE LINUX Enterprise Server 10 for AMD64/EM64T
• SUSE LINUX Enterprise Server 10 with Xen for AMD64/EM64T
• SUSE LINUX Enterprise Server 11 for AMD64/EM64T
• SUSE LINUX Enterprise Server 11 with Xen for AMD64/EM64T
• VMware ESX 4.1
• VMware ESXi 4.1
• VMware vSphere 5

The EN4054 4-port 10Gb Ethernet Adapter supports the following operating systems:

• AIX Version 6.1
• AIX Version 7.1
• IBM i 6.1
• IBM i 7.1
• SUSE LINUX Enterprise Server 11 for IBM POWER
• Red Hat Enterprise Linux 6 for IBM POWER

Support for operating systems is based on the combination of the expansion card and the blade server on which it is installed. See the IBM ServerProven website for the latest information about the specific versions and service packs supported. Select the blade server, and then select the expansion card to see the supported operating systems:http://www.ibm.com/systems/info/x86servers/serverproven/compat/us/


Warranty

There is a 1-year, customer-replaceable unit (CRU) limited warranty. When installed in a server, these adapters assume your system’s base warranty and any IBM ServicePac® upgrade.


Physical specifications

The dimensions and weight of the adapter are as follows:

• Width: 100 mm (3.9 inches)
• Depth: 80 mm (3.1 inches)
• Weight: 13 g (0.3 lb)

Shipping dimensions and weight (approximate):

• Height: 58 mm (2.3 in)
• Width: 229 mm (9.0 in)
• Depth: 208 mm (8.2 in)
• Weight: 0.4 kg (0.89 lb)

Regulatory compliance

The adapter conforms to the following standards:

• United States FCC 47 CFR Part 15, Subpart B, ANSI C63.4 (2003), Class A
• United States UL 60950-1, Second Edition
• IEC/EN 60950-1, Second Edition
• FCC - Verified to comply with Part 15 of the FCC Rules, Class A
• Canada ICES-003, issue 4, Class A
• UL/IEC 60950-1
• CSA C22.2 No. 60950-1-03
• Japan VCCI, Class A
• Australia/New Zealand AS/NZS CISPR 22:2006, Class A
• IEC 60950-1(CB Certificate and CB Test Report)
• Taiwan BSMI CNS13438, Class A
• Korea KN22, Class A; KN24
• Russia/GOST ME01, IEC-60950-1, GOST R 51318.22-99, GOST R 51318.24-99, GOST R 51317.3.2-2006, GOST R 51317.3.3-99
• IEC 60950-1 (CB Certificate and CB Test Report)
• CE Mark (EN55022 Class A, EN60950-1, EN55024, EN61000-3-2, EN61000-3-3)
• CISPR 22, Class A

Popular configurations

The adapters can be used in various configurations. The following figure shows CN4054 10Gb Virtual Fabric Adapters installed in both slots of the x240 (a model without the Embedded 10Gb Virtual Fabric Adapter), which in turn is installed in the chassis. The chassis also has four IBM Flex System Fabric EN4093 10Gb Scalable Switches, each with Upgrade 1 installed to enable 28 internal ports.


Figure 4. Example configuration

The following table lists the parts that are used in the configuration.

Table 6. Components used when connecting the adapter to the 10 GbE switches

Part number/machine type	Description	Quantity
8737-x1x	IBM Flex System x240 Compute Node or other supported server (without Embedded 10Gb Virtual Fabric Adapter)	1 to 14
90Y3554	IBM Flex System CN4054 10Gb Virtual Fabric Adapter	2 per server
8721-A1x	IBM Flex System Enterprise Chassis	1
49Y4270	IBM Flex System Fabric EN4093 10Gb Scalable Switch	4
49Y4798	IBM Flex System Fabric EN4093 10Gb Scalable Switch (Upgrade 1)	4

Related publications

For more information refer to the following resources:

• IBM Flex System Fabric EN4093 10Gb Scalable Switch Product Guide
  http://www.redbooks.ibm.com/abstracts/tips0864.html
• IBM Flex System EN4091 10Gb Ethernet Pass-thru Product Guide
  http://www.redbooks.ibm.com/abstracts/tips0865.html
• IBM Flex System EN2092 1Gb Ethernet Scalable Switch Product Guide
  http://www.redbooks.ibm.com/abstracts/tips0861.html
• IBM Flex System x240 Compute Node Product Guide
  http://www.redbooks.ibm.com/abstracts/tips0860.html
• IBM Flex System p260 and p460 Compute Node Product Guide
  http://www.redbooks.ibm.com/abstracts/tips0880.html
• IBM Redbooks Product Guides for IBM Flex System servers and options
  http://www.redbooks.ibm.com/portals/puresystems
• IBM Redbooks® publication IBM Flex System Products and Technology, SG24-7984
  http://www.redbooks.ibm.com/abstracts/sg247523.html
• IBM Flex System Interoperability Guide
  http://ibm.com/support
• Configuration and Option Guide
  http://www.ibm.com/systems/xbc/cog/
• Standalone Solutions Configuration Tool (SSCT )
  http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-62168
• IBM System x and Cluster Solutions configurator (x-config)
  https://ibm.com/products/hardware/configurator/americas/bhui/asit/install.html
• IBM Configurator for e-business (e-config)
  http://ibm.com/services/econfig/