BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter

The BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter offers the most bandwidth of any blade switch and represents the perfect migration platform for clients who are still at 1 Gb outside the chassis by seamlessly integrating in the existing 1 Gb infrastructure. This is the first 10 Gb switch for IBM BladeCenter that is convergence ready (that is, able to transmit Converged Enhanced Ethernet (CEE) to a Fibre Channel over Ethernet (FCoE ) capable top-of-rack switch). This new feature is available with firmware release 6.1.

In addition, this switch is a key part of the IBM Virtual Fabric offering, which allows clients to form eight virtual network interface controllers (NICs) from one physical NIC and to manage them in virtual groups. This switch can be managed via a command-line interface (CLI) or a graphical interfaces of the switch and in the future with BladeCenter Open Fabric Manager, providing all the benefits of I/O Virtualization at 10 Gb speeds. Figure 1 shows the switch module.

Figure 1. BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter

Did you know

As virtualization has become a prevalent standard in the data center, more people have been looking for ways to virtualize I/O to reduce cost and complexity while also maximizing I/O resources. Virtual Fabric for IBM BladeCenter provides fast, flexible, easy, and reliable I/O using the BNT 10G Ethernet Switch Module and the Emulex Virtual Fabric Adapter, which provides the abilities to carve up an NIC into eight virtual NICs, dynamically allocate bandwidth, manage virtual groups, and so on.

The BNT switch is designed to support a number of different types of configurations from the blade: 1 Gb, 10 Gb, virtual NIC, Converged Enhanced Ethernet (CEE/FCoE), and iSCSI. If you have a chassis with multiple servers, some operating at 1 Gb, some at 10 Gb, and some transmitting converged packets, this single switch can handle all these workloads and can connect to a 1 Gb infrastructure or a 10 Gb infrastructure, or both.


Part number information

Table 1 shows the part numbers for ordering these modules and additional options for them.

Table 1. IBM part numbers and feature codes for ordering

Description	IBM part number	IBM feature code
BNT Virtual Fabric 10Gb Switch Module*	46C7191	1639
IBM 10GBase-SR 10GbE 850 nm Fiber SFP+ Transceiver	44W4408	4942

* Formerly known as the BNT 10-port 10Gb Ethernet Switch Module

The module part numbers include the following items:

• One BNT Virtual Fabric 10Gb Switch Module
• 3-meter mini-USB-to-DB9 serial console cable
• One filler module
• BNT Virtual Fabric 10Gb Switch Module Installation Guide
• BNT user license agreement
• Important Notices document
• Documentation CD-ROM

Note: SFP+ (small form-factor pluggable plus) transceivers are not included.

With the flexibility of the BNT switch, clients can take advantage of the technologies they require for multiple environments. For 1 Gb uplinks, they can take advantage of SFP transceivers. For 10Gb uplinks, they have a choice of either SFP+ transceivers (SR or LR) for longer distances or more cost effective and lower power consuming options such as SFP+ direct-attached cables (DAC), which can be 1-7 meters in length and are ideal for connecting chassis together, connecting to a top-of-rack switch, or even connecting to an adjacent rack.

Table 2 shows additional transceivers and direct-attach cable options that are available directly from BNT. These are in addition to the 10GBase-SR SFP+ transceiver available from IBM listed in Table 1.

Table 2. BNT part numbers for ordering SFP-based transceivers and cables

Description	BNT part number
BLADE 1000Base-T SFP transceiver	BN-CKM-S-T
BLADE 1000Base-SX SFP transceiver	BN-CKM-S-SX
BLADE 10GBase-LR SFP+ transceiver	BN-CKM-SP-LR
SFP+ Copper Direct Attach Cable - 1M	BN-SP-CBL-1M
SFP+ Copper Direct Attach Cable - 3M	BN-SP-CBL-3M
SFP+ Copper Direct Attach Cable - 7M	BN-SP-CBL-7M

Benefits

The BNT Virtual Fabric 10Gb Switch Module offers the following benefits:

• Integration and consolidation: This module offers integration within the BladeCenter chassis, consolidating full Layer 2-3 LAN switching and routing capabilities into a single chassis. This helps flatten the data center infrastructure and reduces the number of discrete devices, management consoles, and equipment that administrators must deal with, helping lower costs and simplifying deployment.
• Cost: The BNT switch is priced extremely competitively compared with external switches, especially when you factor in that no cables are required between the blade and the switch. In addition, the ability to use direct-attach cables with the switch can help clients save even more compared with the more expensive CX4, XFP, or X2 transceivers.
• Performance: With support for ten 10 Gb uplinks, clients can exploit not only up to 200 Gbps of bi-directional uplink bandwidth, but also an extremely low oversubscription (14 to 10), which can support even the most performance-intensive environments (up to 7.2 Gbps per blade server port). Clients wanting extreme performance can use up to four switches and the quad port 10 Gb adapter and get up to 1.92 Tbps of data per BladeCenter H chassis.
• Lower power consumption: BNT using only 75 W per switch delivers extreme performance per watt, which is second to none with its support for up to 6.4 Gbps per watt of power.
• Layer 3 functionality: The BNT switch module includes Layer 3 functionality, which provides security and performance benefits as inter-VLAN traffic stays within the chassis. This switch also provides the full range of Layer 3 protocols from static routes for technologies such as Open Shortest Path First (OSPF) and Border Gateway Protocol (BGP) for enterprise customers.
• Interoperability: BNT switches interoperate seamlessly with other vendors' upstream switches.
• Management: These switches are designed to support multiple CLIs, allowing the IT staff to select the CLI with which they are most comfortable. Choices are BLADEOS CLI for those with a Nortel switch background, industry-based CLI (Cisco-like) for those familiar with IOS, and a full-function Web-based GUI for the latest in simplicity.
• Fault tolerance: These BNT switches learn alternate routes automatically and perform faster convergence in the unlikely case of a link, switch, or power failure. The switch uses proven technologies like L2 trunk failover, advanced VLAN-based failover, VRRP, IGMP V3 snooping, and OSPF.
• Converged fabric: The BNT switch is designed to support CEE and connectivity to FCoE modules, making the IBM solution fully FCoE capable. CEE will help enable clients to combine storage, messaging traffic, VoIP, video, and other data on a common data center Ethernet infrastructure. FCoE will help enable highly efficient block storage over Ethernet for consolidating server network connectivity. As a result, clients can deploy a single server interface for multiple data types, which can simplify both deployment and management of server network connectivity, while maintaining the high availability and robustness required for storage transactions.

The BNT switch comes with a three-year software license that allows for the download of new features, such as:

• Virtual Fabric for IBM BladeCenter: This new firmware allows for clients to implement and manage the Virtual Fabric solution via a graphical interface or CLI. It is through this that administrators can perform tasks such as activating a Virtual Fabric, selecting the number of virtual NICs (vNICs), dynamically allocating bandwidth, creating virtual groups to simplify management, and setting up the failover and isolation capabilities to the vNICs for better availability and security.
• SmartConnect with VMready: VMready is a unique solution that enables the network to be virtual machine-aware. The network can be configured and managed for virtual ports (v-ports) rather than just for physical ports. With VMready, as VMs migrate across physical hosts, so do their network attributes. Virtual machines can be added, moved, and removed while retaining the same ACLs, QoS, and VLAN attributes. VMready allows for a define-once-use-many configuration that evolves as the server and network topologies evolve. VMready works with all virtualization products, including VMware, Hyper-V, Xen, and KVM, without modification of virtual machine hypervisors or guest operating systems.

Features and specifications

The BNT Virtual Fabric 10Gb Switch Module includes the following features and functions:

• Form-factor
  • Single-wide high-speed switch module
• Internal ports
  • 14 internal auto-negotiating ports: 1 Gb or 10 Gb to the server blades
  • Two internal full-duplex 100 Mbps ports connected to the management module
• External ports
  • Up to ten 10 Gb SFP+ ports (also designed to support 1 Gb SFP if required, flexibility of mixing 1 Gb/10 Gb)
  • One 10/100/1000 Mb copper RJ-45 used for management or data
  • An RS-232 mini-USB connector for serial port that provides an additional means to install software and configure the switch module
• Scalability and performance
  • Autosensing 1 Gb/10 Gb internal and external Ethernet ports for bandwidth optimization
  • Non-blocking architecture with wire-speed forwarding of traffic and full line rate performance of 480 Gbps full duplex
  • Media access control (MAC) address learning: automatic update, supports up to 32 Kb MAC addresses
  • Up to 128 IP interfaces per switch
  • Static, EtherChannel, and LACP (IEEE 802.3ad) link aggregation, up to 100 Gb of total bandwidth per switch, up to 18 trunk groups, and up to eight ports per group
  • Support for jumbo frames (up to 12288 bytes)
  • Broadcast/multicast storm control
  • IGMP snooping for limit flooding of IP multicast traffic (IGMP V1, V2, and V3)
  • Configurable traffic distribution schemes over trunk links based on source/destination IP addresses, MAC addresses, or both
  • Fast port forwarding and fast uplink convergence for rapid STP convergence
• Availability and redundancy
  • VRRP for Layer 3 router redundancy
  • IEEE 802.1D STP for providing Layer 2 redundancy with PVRST+
  • IEEE 802.1s Multiple STP (MSTP) for topology optimization, up to 128 STP instances supported by single switch
  • IEEE 802.1w Rapid STP (RSTP), providing rapid STP convergence for critical delay-sensitive, traffic-like voice or video
  • Layer 2 Trunk Failover to support active/standby configurations of network adapter teaming on blades
  • Interchassis redundancy (Layer 2 and Layer 3)
• Converged Enhanced Ethernet
  • CEE support and connectivity to FCoE
  • Connects to a top-of-rack switch for connectivity to a Fibre Channel SAN
• VLAN support
  • Up to 1024 VLANs supported per switch; VLAN numbers ranging from 1 to 4095 (4095 is used for the management module’s connection only.)
  • 802.1Q VLAN tagging support on all ports
  • Protocol-based VLANs
• Security
  • VLAN-based, MAC-based, and IP-based access control lists (ACLs)
  • 802.1X port-based authentication
  • Multiple user IDs and passwords
  • User access control
  • Radius, TACACS+, LDAP
• Quality of service (QoS)
  • Up to eight queues per port
  • Support for IEEE 802.1p, IP ToS/DSCP, and ACL-based (MAC/IP source and destination addresses, VLANs) traffic classification and processing
  • Traffic shaping and re-marking based on defined policies
  • Eight weighted round robin (WRR) priority queues per port for processing qualified traffic
• Layer 3 functions
  • IP forwarding
  • IP filtering with ACLs (up to 4096 ACLs supported)
  • VRRP for router redundancy
  • Support for up to 128 static routes
  • Routing protocol support (Router Information Protocol (RIP) V1; RIP V2; OSPF V1, V2, and V3; BGP-4), up to 1024 entries in routing table
  • Support for DHCP Relay
  • IPv6 management (full IPv6 support including routing is planned to be released in optional version of firmware in 2009)
• Manageability
  • Simple Network Management Protocol (SNMP, V1, V2, and V3)
  • HTTP/HTTPS browser GUI
  • Industry-standard CLI and BLADEOS/AlteonOS CLI
  • Telnet interface for CLI
  • SSH V1/V2
  • Serial interface for CLI
  • Scriptable CLI
  • Firmware image update (TFTP and FTP)
  • Network Time Protocol (NTP) for switch clock synchronization
  • BNT BLADEHarmony Manager support
• Monitoring
  • Switch LEDs for external port status and switch module status indication
  • Port mirroring for analyzing network traffic passing through switch
  • Change tracking and remote logging with syslog feature
  • POST diagnostics
• Special functions
  • Serial over LAN (SOL)
• Future features
  • SmartConnect with VMready, firmware update planned for 2H/2009

The switch module supports the following IEEE standards:

• IEEE 802.1D STP with PVRST+
• IEEE 802.1s MSTP
• IEEE 802.1w RSTP
• IEEE 802.1p Tagged Packets
• IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled)
• IEEE 802.1x port-based authentication
• IEEE 802.2 Logical Link Control
• IEEE 802.3ad Link Aggregation Control Protocol
• IEEE 802.3x Full-duplex Flow Control
• IEEE 802.3ab 1000BASE-T Gigabit Ethernet
• IEEE 802.3ae 10GBASE-SR 10Gb Ethernet fiber optics short range
• IEEE 802.3ae 10GBASE-LR 10Gb Ethernet fiber optics long range
• IEEE 802.3z 1000BASE-SX Gigabit Ethernet

Supported BladeCenter chassis and expansion cards

The BNT Virtual Fabric 10Gb Switch Module is supported in the IBM BladeCenter H and HT chassis listed in Table 3.

Table 3. IBM BladeCenter chassis that supports the BNT Virtual Fabric 10Gb Switch Module

I/O module	Part number
BNT Virtual Fabric 10Gb Switch Module	46C7191	N	N	Y	N	Y	N	N

The BNT Virtual Fabric 10Gb Switch Module supports the expansion cards listed in Table 4. Table 4 also lists the chassis bays in which the switch module must be installed when used with each expansion card.

Table 4. BNT Virtual Fabric 10Gb Switch Module and BladeCenter chassis I/O bays support

Description	Part number
Gigabit Ethernet integrated in the server	None	N	N	N	N	N	N	N	N	N	N
Ethernet Expansion Card (CFFv)	39Y9310	N	N	N	N	N	N	N	N	N	N
Ethernet Expansion Card (CIOv)	44W4475	N	N	N	N	N	N	N	N	N	N
QLogic Ethernet and 4Gb FC Card (CFFh)	39Y9306	N	N	N	N	N	N	N	N	N	N
2/4 Port Ethernet Expansion Card (CFFh)	44W4479	N	N	N	N	N	N	Y	Y	Y	Y
QLogic Ethernet and 8Gb FC Card (CFFh)	44X1940	N	N	N	N	N	N	N	N	N	N
NetXen 10Gb Ethernet Expansion Card (CFFh)	39Y9271	N	N	N	N	N	N	Y	N	Y	N
Broadcom 10Gb 2-port Ethernet Exp. Card (CFFh)	44W4466	N	N	N	N	N	N	Y	N	Y	N
Broadcom 10Gb 4-port Ethernet Exp. Card (CFFh)	44W4465	N	N	N	N	N	N	Y	Y	Y	Y
Broadcom 10 Gb Gen 2 2-port Ethernet Exp. (CFFh)	46M6168	N	N	N	N	N	N	Y	N	Y	N
Broadcom 10 Gb Gen 2 4-port Ethernet Exp. (CFFh)	46M6164	N	N	N	N	N	N	Y	Y	Y	Y
QLogic 2-port 10Gb Converged Network Adapter (CFFh)	42C1830	N	N	N	N	N	N	Y*	N	Y*	N
Emulex Virtual Fabric Adapter (CFFh)	49Y4245	N	N	N	N	N	N	Y	N	Y	N

* Support is planned.

The BladeCenter chassis has the following bays:

• BladeCenter S, E, and T have four standard I/O bays (1, 2, 3, and 4).
• BladeCenter H has four standard I/O bays (1, 2, 3, and 4), two bridge bays (5 and 6), and four high-speed bays (7, 8, 9, and 10).
• BladeCenter HT has four standard I/O bays (1, 2, 3, and 4) and four high-speed bays (7, 8, 9, and 10).

The BNT Virtual Fabric 10Gb Switch Module fits in one of the high-speed I/O bay (bays 7-10).


Popular configurations

The BNT Virtual Fabric 10Gb Switch Module can be used in various configurations.

Basic two-port configuration

Figure 2 shows the use of BNT Virtual Fabric 10Gb Switch Modules to route two Ethernet ports from Broadcom 2-port 10Gb Ethernet Expansion Cards (CFFh) installed on each server. Two BNT Virtual Fabric 10Gb Switch Modules are installed in bay 7 and bay 9 of the BladeCenter H chassis. All connections between the controller, card, and switch modules are internal to the chassis. No cabling is needed.


Figure 2. A 20 Gb solution using two BNT Virtual Fabric 10Gb Switch Modules

Table 5 lists the components used in this configuration.

Table 5. Components used when connecting Broadcom 2-port 10Gb Ethernet Expansion Card (CFFh) to four BNT 10Gb High Speed Switch Modules

Diagram reference	Part number/machine type	Description	Quantity
	Varies	IBM BladeCenter HS22 or other supported server	1 to 14
	44W4466	Broadcom 2-port 10Gb Ethernet Expansion Card (CFFh)	1 per server
	8852 or 8740/8750	BladeCenter H or BladeCenter HT	1
	46C7191	BNT Virtual Fabric 10Gb Switch Module	2
	44W4408	IBM 10GBase-SR SFP+ Transceiver	Up to 20*

*The BNT 10Gb High Speed Switch Module has 10 external 10 Gb ports. To communicate outside of the chassis, you must have either one SFP+ transceiver or SFP+ direct-attached cable (DAC) connected. You have the flexibility to expand your bandwidth as you see fit using from one to 10 connections per switch.

4-port configuration

Figure 3 shows the use of BNT Virtual Fabric 10Gb Switch Modules to route four Ethernet ports from a Broadcom 4-port 10Gb Ethernet Expansion Card (CFFh) installed on each server. Four BNT Virtual Fabric 10Gb Switch Modules are installed in bay 7, bay 8, bay 9, and bay 10 of the BladeCenter H chassis. All connections between the controller, card, and switch modules are internal to the chassis. No cabling is needed.


Figure 3. A 40 Gb solution using four BNT Virtual Fabric 10Gb Switch Modules

Table 6 lists the components used in this configuration.

Table 6. Components used when connecting Broadcom 4-port 10Gb Ethernet Expansion Card (CFFh) to four BNT 10Gb High Speed Switch Modules

Diagram reference	Part number/machine type	Description	Quantity
	Varies	IBM BladeCenter HS22 or other supported server	1 to 14
	44W4465	Broadcom 4-port 10Gb Ethernet Expansion Card (CFFh)	1 per server
	8852 or 8740/8750	BladeCenter H or BladeCenter HT	1
	46C7191	BNT Virtual Fabric 10Gb Switch Module	4
	44W4408	IBM 10GBase-SR SFP+ Transceiver	Up to 40*

*The BNT 10Gb High Speed Switch Module has 10 external 10 Gb ports. In order to communicate outside of the chassis you must have either one SFP+ transceiver or SFP+ direct-attached cable (DAC) connected. You have the flexibility to expand your bandwidth as you see fit using from one to 10 connections per switch.


Connectors and LEDs

Figure 4 shows the front panel of the BNT Virtual Fabric 10Gb Switch Module.


Figure 4. Front panel of the BNT Virtual Fabric 10Gb Switch Module

The front panel contains the following components:

• LEDs display the status of the switch module and the network:
  • OK: indicates that the switch module has passed the power-on self-test (POST) with no critical faults and is operational
  • Switch-module error: indicates that the switch module has failed the POST or detected an operational fault
• One USB RS-232 console port provides an additional means to install software and configure the switch module. This USB-style connector enables the connection of a special serial cable that is supplied with the switch module.
• Ten SFP+ port connectors to attach SFP+ modules.
• One RJ-45 Ethernet port connector.
• Each external SFP+ port on the switch module contains a link/activity LED, and an RJ-45 port contains Ethernet link OK and Ethernet Tx/Rx activity LEDs.

Network cabling requirements

The following network cables are supported for the BNT Virtual Fabric 10Gb Switch Module:

• 10GBASE-SR for 10 Gb ports: 850 Nm wavelength, multimode fiber, 50 µ or 62.5 µ (300 meters maximum), with LC duplex connector
• 1000BASE-T for RJ-45 port:
  • UTP Category 6 (100 meters maximum)
  • UTP Category 5e (100 meters maximum)
  • UTP Category 5 (100 meters maximum)
  • EIA/TIA-568B 100-ohm STP (100 meters maximum)

Related publications

For more information see the following BNT Virtual Fabric 10Gb Switch Module product publications, available from:

http://www.ibm.com/support/docview.wss?uid=psg1MIGR-5080917

• Installation Guide
• Application Guide
• Command Reference
• isCLI Reference
• BBI (Browser-based Interface) Quick Guide
• Release Notes
• End User License Agreement

Other documents:

• IBM BladeCenter Virtual Fabric home page:
  http://www.ibm.com/systems/bladecenter/hardware/openfabric/virtualfabric.html
• IBM US Announcement Letter:
  http://ibm.com/common/ssi/cgi-bin/ssialias?infotype=dd&subtype=ca&&htmlfid=897/ENUS109-288
• IBM BladeCenter Interoperability Guide:
  http://www.ibm.com/support/docview.wss?uid=psg1MIGR-5073016
• IBM Redbooks publication IBM BladeCenter Products and Technology, SG24-7523:
  http://www.redbooks.ibm.com/abstracts/sg247523.html