IBM DS8870 - High-Performance Flash Enclosure

IBM Redbooks Product Guide


Abstract

This IBM® Redbooks® product guide describes the DS8870 high-performance flash enclosure (HPFE).

Contents


The high-performance flash enclosure (HPFE) is a Redundant Array of Independent Disks (RAID) storage enclosure that can support up to thirty 400 GB encryption capable flash cards (1.8-inch, 46 mm form factor) in a 1U rack space. The high-performance flash enclosures and flash cards can only be installed in the DS8870 base frame.

The DS8870 Release 7.3 firmware can support the following configurations:
  • The DS8870 Enterprise Class or Business Class configurations can support up to four HPFEs in a base frame, along with up to 10 standard drive enclosures.
  • The DS8870 High Performance All Flash configuration can contain up eight HPFEs (four vertically and four horizontally) in a single base frame, for a maximum of 240 flash cards. No standard drive enclosures can be added.

Figure 1 shows the IBM® DS8870 high-performance flash enclosure.


Figure 1. Perspective view of HPFE


Did you know?

Compared to the flash drives (also known as solid-state drives or SSDs) that were already available for the DS8870, the high-performance flash enclosure provides a new and higher standard of flash performance. As implemented in a DS8870, the HPFE is directly attached to the PCIe fabric, enabling increased bandwidth compared to standard Fibre Channel-attached storage enclosures.

Flash cards are high-input/output operations per second (IOPS) class enterprise storage devices that are targeted at I/O-intensive workload applications that can benefit from a high level of fast-access storage.

Flash cards along with flash drives are considered as part of Tier 0. Both types offer a number of potential benefits over spinning drives, including better IOPS, lower power consumption, less heat generation, and lower acoustical noise. However, the flash cards offer even higher throughput using the flash RAID adapters in the high-performance flash enclosure and direct PCIe connections to the processor complexes.

The IBM Easy Tier® intra-tiering auto-rebalance (micro-tiering) feature is used to distribute the workload among traditional flash drives and flash cards according to their IOPS capacity within the highest storage tier (Tier 0).


High-performance flash enclosure highlights

The high-performance flash enclosure comes with various capabilities and performance characteristics, as summarized here:
  • The high-performance flash enclosure delivers up to 250,000 IOPS and up to 3.4 GBps bandwidth in just one EIA (1U) of rack space.
  • Using high-performance flash enclosures can accelerate database performance by up to four times while still guaranteeing the industry-leading DS8870 availability.
  • By using high-performance flash enclosures, batch times shrink by up to 10%, therefore reducing impact on production applications.
  • High-performance flash enclosures can carry up to 12 TB of raw capacity per enclosure.
  • Flash cards in the high-performance flash enclosure support full drive encryption.


Architecture and key components

The high-performance flash enclosure is a 1U enclosure that integrates a fully redundant pair of flash RAID adapters, dual power supplies with integrated cooling fans and up to thirty 1.8-inch encryption capable flash cards.

The flash RAID adapters are directly connected over a PCIe fabric to the DS8870 I/O enclosures, with a Gen2 PCIe cable. Each connection provides a 2 GBps full duplex connection.

Effectively, the high-performance flash enclosure combines the RAID device adapter into the storage enclosure. The flash RAID adapter is a device adapter much like the device adapter in traditional Fibre Channel Arbitrated Loop (FC-AL) topology. The flash RAID adapter includes an SAS expander providing enclosure management services.

Each high-performance flash enclosure contains the following hardware components:
  • 16 or 30 400-GB 1.8-inch SAS flash cards, which support IBM Full Disk Encryption (FDE).
    Figure 2 shows the 1.8-inch SAS flash cards.

    Figure 2. 1.8-inch flash cards assembly
    If the high-performance flash enclosure is ordered with only 16 flash cards, 14 flash card fillers are installed in the empty flash card locations. When 14 additional flash cards are installed, to fully populate the enclosure, the fillers are removed.

    Note: Flash cards are not available for capacity on demand (CoD) configurations.

    The flash cards are installed into the HPFE in rows, starting from its upper left side (in case of horizontal installation). See Figure 3.


    Figure 3. Front view of high-performance flash enclosure in horizontal positions
  • Two power supplies with integrated cooling fans, as shown in Figure 4.

    Figure 4. Two HPFE power supply units
  • Each HPFE contains two flash RAID adapters that provide redundant data paths to the flash cards in the flash enclosure. These adapters also provide enclosure services to control power, cooling, and other non-data functions.
    The dual integrated RAID controllers in the DS8870 high-performance flash enclosures are connected to the I/O Enclosures via PCIe cable. Each cable supports a 2 GB bandwidth (bidirectional), with none of the protocol overhead associated with Fibre Channel architecture. Each I/O enclosure pair supports up to two HPFEs. Figure 5 shows the rear view of the HPFE.


    Figure 5. Rear view of the HPFE
  • One back plane for plugging components. Figure 6 shows where the components are located.

    Figure 6. Back plane assembly

Arrays and spares

The high-performance flash enclosures can each contain up to four array sites. The first group of 16 Flash Cards (required) in each flash enclosure will form two 8-flash card array sites. Each of these two array sites will designate one flash card as a spare card. The second set of 14 cards (optional) will form two 7-card array sites and no other spare will be created, for a total of two spares per flash enclosure. This configuration is unique to the high-performance flash enclosure. This is a change from the array logical configuration process used with standard drive enclosures, where all array sites have eight drives.

During the configuration process, arrays are created from the array sites. The flash enclosures support only RAID 5 arrays, which will be either 6+P+S or 6+P.

Configurations

This section gives an overview of the possible configurations for DS8870 including the high-performance flash enclosure.

Figure 7 shows a table with all configurations possible for DS8870, highlighting the maximum number of flash cards, flash drives, and spinning disks.


Figure 7. Table showing DS8870 possible configurations

The high-performance flash enclosure is available in the following configurations, which are depicted in Figure 8:
  • DS8870 Model 961 Enterprise Class or Business Class configuration:
    In the Enterprise or Business Class configuration, you can have up to four HPFEs installed vertically in the base frame. Each HPFE can have either 16 or 30 flash cards. The base frame can still accommodate up to 10 standard drive enclosures, populated with either flash drives (SSDs) or spinning drives.
  • DS8870 Model 961 High Performance All Flash configuration
    In the High Performance All Flash configuration, up to 8 HPFEs (four horizontally and four vertically) can be installed in the base frame. A High Performance All Flash configuration has four I/O enclosure pairs installed expanding the number of hosts and maximizing the bandwidth with up to 16 four or eight port Fibre Channel/IBM FICON® host adapters, offering maximum performance and connectivity in a single frame footprint.
    The High Performance All Flash configuration does not support any standard disk enclosures or expansion frames.


    Figure 8. DS8870 base frame with HPFEs
For more information, see the IBM DS8870 Architecture and Implementation, SG24-8085 IBM Redbooks publication.


Connectivity

The high-performance flash enclosures are directly attached to the I/O enclosure by using a PCIe connection, which increases bandwidth and transaction-processing capability compared to standard drive enclosures connectivity (FC-AL). The flash interface cards extend the I/O enclosure PCIe connections to the flash enclosure. Each flash interface card is a PCIe redriver card that plugs into the I/O enclosure.

Figure 9 is a block diagram that shows a simplified view of the PCIe cabling topology.


Figure 9. High-performance flash enclosure schematic cabling

Figure 10 shows the locations for flash interface adapters in the DS8870 I/O enclosure, along with the host adapters (HAs) and standard enclosure device adapters (DAs).


Figure 10. Flash interface adapters are shown in the I/O enclosure

Population order and plug order

This section explains the plug order of HPFEs in the DS8870 961 frame, and applies whether it is an All Flash, Business Class, or Enterprise Class configuration.

The DA pairs to which the HPFEs are attached are numbered 8 - 15. In Business or Enterprise Class configuration, HPFEs attaching to DA pairs 8 - 11 are installed vertically in the following sequence, 10->8->11->9, while in High Performance All Flash configurations, HPFEs attaching to DA pairs 8 - 15 are installed in the sequence 10->8->11->9->14->12->15->13.

Each HPFE is connected by its two flash RAID adapters to two different Flash Interface Cards (FICs), each of them in a different I/O bay.

Figure 11 (for Business Class and Enterprise Class) and Figure 12 (for All Flash) show HPFE connections and plug order with colors and numbers that indicate the corresponding connected Flash RAID adapters and FICs.


Figure 11. Enterprise class HPFEs plug order (only base frame shown)


Figure 12. All Flash system HPFEs plug order


Easy Tier and flash cards

IBM Easy Tier, a well-proven DS8870 feature is available at no charge. It dynamically optimizes performance for multi-tiered systems. It can also rebalance data within a single tier to help maintain optimal performance. Now in its sixth generation, Easy Tier offers full support for the high-performance flash enclosure, while maintaining additional features like Easy Tier Server, Easy Tier Application, and Easy Tier Heat Map Transfer.

Note: All previously announced Easy Tier functions apply, including manual volume migration, automated sub-volume data relocation, automated performance rebalancing within drive tiers (in both single tier and multi-tier drive pools), hotspot management, and rank depopulation.

Currently, IBM supports up to three drive classes or storage tiers that can be configured in the same DS8870:
  • Tier 0 or Flash tier: Contains flash cards and flash drives. Although flash cards are in the same tier as flash drives (SSDs), the Easy Tier intra-tier auto-rebalance function can exploit the higher IOPS capability of the HPFE.
  • Tier 1 or Enterprise tier: Contains the Enterprise drives (SAS 15 k or 10 k rpm).
  • Tier 2 or Nearline tier: Contains the Nearline drives (SAS 7.2 k rpm).

For more information, see the IBM DS8870 Easy Tier, REDP-4667 IBM Redbooks publication.


Disk Magic

Disk Magic is a Windows based disk subsystem performance modeling tool that is used by IBM and IBM Business Partners to model disk storage subsystem performance. It supports disk systems from multiple vendors and offers detailed support for IBM storage systems. Contact your IBM Representative or IBM Business Partner to evaluate a Disk Magic study.

Disk Magic supports the high-performance flash enclosure in the DS8870 with Licensed Machine Code (LMC) 7.7.30.xx.xx (bundle version 87.30.xx.xx) or later.


Technical specifications summary

Table 1 summarizes the high performance flash enclosure specifications.

Table 1. DS8870 high-performance flash enclosure specifications
ModelDS8870 (961)
Size19 inch
Communication ProtocolPCIe
Maximum Quantity in a system8 pcs (4 pcs in Business or Enterprise Classes)
Flash Card interface6 Gbps SAS Flash Drive
Number of disk drives (minimum/maximum)16/30
Maximum physical storage capacity12 TB (9.11 TB usable capacity, RAID 5)
Disk sizes400 GB
RAID levels5

For more information, see the IBM DS8870 Introduction and Planning Guide, GC27-4209 IBM publication.


Upgrades

The IBM DS8870 high-performance flash enclosures are only supported in the DS8870 with Licensed Machine Code (LMC) 7.7.30.xx.xx.

Conversion of Business class or Enterprise configuration into a High Performance All Flash configuration is not supported.


Related publications and links

For more information, see the following links:

Special Notices

This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment. publib-b.boulder.ibm.com

Profile

Publish Date
20 May 2014

Last Update
29 May 2014


Rating: Not yet rated


Author(s)

IBM Form Number
TIPS1181