IBM DS8870 - High-Performance Flash Enclosure

IBM Redbooks Product Guide


Abstract

This IBM® Redbooks® product guide describes the DS8870 high-performance flash enclosure (HPFE).

Contents


The high-performance flash enclosure (HPFE) is a Redundant Array of Independent Disks (RAID) storage enclosure that can support sixteen or thirty 400 GB encryption capable flash cards (1.8-inch, 46 mm form factor) in a 1U rack space.

HPFEs can be installed in the IBM DS8870 storage system. As of DS8870 microcode Release 7.4:
  • One to eight HPFEs can be installed in the DS8870 High-Performance All-Flash configuration
  • Up to four HPFEs can be installed in the DS8870 Enterprise Class and Business Class configuration base frame, and up to four HPFEs in the first expansion frame

Figure 1 shows the IBM® DS8870 high-performance flash enclosure.


Figure 1. Perspective view of HPFE


Did you know?

Compared to the flash drives (also known as solid-state drives or SSDs) that were already available for the DS8870, the high-performance flash enclosure provides a higher standard of flash performance. As implemented in the DS8870, the HPFE is directly attached to the PCIe fabric, enabling increased bandwidth compared to standard Fibre Channel-attached storage enclosures.

Flash cards are high-input/output operations per second (IOPS) class enterprise storage devices that are targeted at I/O-intensive workload applications that can benefit from a high level of fast-access storage.

High Performance Flash cards, along with flash drives, are considered as Tier 0 storage. Both offer a number of potential benefits over spinning drives, including better IOPS, lower power consumption, less heat generation, and lower acoustical noise. However, the flash cards offer even higher throughput using the flash RAID adapters in the high-performance flash enclosure and direct PCIe connectivity to the processor complexes, compared to the fiber attached flash drives installed in the standard drive enclosures.

The IBM Easy Tier® intra-tiering auto-rebalance feature is used to distribute the workload among traditional flash drives and flash cards according to their IOPS capacity within the highest storage tier (Tier 0).

High-performance flash enclosure highlights

The high-performance flash enclosure comes with various capabilities and performance characteristics, as characterized below:
  • The High-Performance Flash Enclosure delivers up to 250,000 IOPS and up to 3.4 GBps bandwidth in just one EIA (1U) of rack space.
  • Database performance can be accelerated by up to four times
  • Batch times can shrink by up to 10%, therefore reducing impact on production applications.
  • Each High-Performance Flash Enclosure contains up to 12 TB of raw capacity.
  • Up to 8 HPFEs per DS8870, for a total of 96 TB raw capacity
  • Flash cards in the High-Performance Flash Enclosure support encryption

Architecture and key components

The high-performance flash enclosure is a 1U enclosure that integrates a fully redundant pair of flash RAID adapters, dual power supplies with integrated cooling fans and sixteen or thirty 1.8-inch encryption capable flash cards.

The flash RAID adapters are directly connected over a PCIe fabric to the DS8870 I/O enclosures, with a Gen2 PCIe cable. Each connection provides a 2 GBps full duplex connection.

Effectively, the high-performance flash enclosure combines the RAID device adapter into the storage enclosure. The flash RAID adapter is a device adapter much like the device adapter in traditional Fibre Channel Arbitrated Loop (FC-AL) topology. The flash RAID adapter includes an SAS expander providing enclosure management services.

The graphic displayed in Figure 2 shows the major components of the HPFE. Top cover and cover from one of the Flash RAID adapters has been removed.


Figure 2. High Performance Flash Enclosure - Key components
Flash RAID adapter

Each HPFE contains two flash RAID adapters that provide redundant data paths to the flash cards in the enclosure. These adapters also provide enclosure services to control power, cooling, and other non-data functions. Each flash RAID adapter has there own redundant cooling fans.

Each flash RAID adapter connects to a DS8870 I/O enclosure pair by a 2 GB PCIe Gen 2 cable. The PCIe connectivity has none of the protocol overhead associated with Fibre Channel architecture. Each I/O enclosure pair supports up to two HPFEs.

Enclosure Power Supplies

Each HPFE has a pair of fully redundant power supply units (PSU). Each poser supply unit has its own integrated fans. Cooling fans from the PSUs and the flash RAID adapters provide cooling for the entire enclosure.

Enclosure Midplane

The enclosure midplane provides the connectivity for the Flash RAID adapters, PSUs, and Flash cards.

High Performance Flash Cards

Each HPFE is installed with a minimum of sixteen 1.8 inch High Performance Flash Cards. An optional second set of 14 flash cards can be installed for a maximum total of thirty flash cards. If the second set is not installed, then a set of fillers would be installed in the empty slots.

The flash cards are installed into the HPFE in rows. See Figure 3 for Flash card locations, when HPFE is installed horizontally and vertically.
    Note: Flash cards are not available for capacity on demand (CoD) configurations.

      Figure 3. Flash card locations within the HPFE

      Virtualization

      The HPFE contains 16 or 30 flash cards. Implemented in the DS8870, HPFE only supports RAID 5 arrays. The HPFE will always have two spare flash cards.
      The arrays are configured as follows;
      • First 16 flash cards installed
          - arraySite 1 = 6+P+S RAID 5
          - arraySite 2 = 6+P+S RAID 5
      • Adding optional 14 flash card set installed
          - arraySite 3 = 6+P RAID 5
          - arraySite 4 = 6+P RAID 5
      Each HPFE can have a maximum of 12 TB of raw capacity, or 9.125 TB of usable capacity when configured with RAID 5. With the supported maximum of eight fully populated HPFEs in a DS8870 storage system, maximum usable RAID 5 capacity is 73 TB of high performance flash.

      High-Performance Flash Enclosures Implemented in the DS8870

      The DS8870 has three configurations:
      • The DS8870 High-Performance All-Flash configuration
        • Supports one to eight HPFEs in a single frame
        • Does not support any standard drive enclosures
        • Offers maximum performance and connectivity in a single frame footprint
      • The DS8870 Enterprise Class configuration
        • Supports up to four HPFEs in the base frame
        • Supports up to four HPFEs in the first expansion frame (Requires Release 7.4 or later)
        • Supports up to 3 expansion frames
        • Supports up to 1536 2.5 inch drives in standard drive enclosures
        • Optimized for performance and is highly scalable, offering a wide range of options for long term growth.
      • The DS8870 Business Class configuration
        • Supports up to four HPFEs in the base frame
        • Supports up to four HPFEs in the first expansion frame (Requires Release 7.4 or later)
        • Supports up to 2 expansion frames
        • Supports up to 1056 2.5 inch drives in standard drive enclosures
        • Increases device adapter utilization, prioritizing cost-effective storage capacity growth.
      Figure 4 shows the HPFE enclosures installed the base frame of the High-Performance All-Flash (LEFT) and Enterprise Class configuration (RIGHT).


      Figure 4. High-Performance All-Flash and Enterprise Class configurations base frames with HPFEs

      Population order and plug order

      The DS8870 High-Performance All-Flash configuration supports a minimum of one HPFE installed in location R1-04.
      Installation order is as follows; R1-B04 , R1-B03 , R1-B02, R1-B01, R1-B14, R1-B13, R1-B12, R1-B11

      Figure 5 shows HPFE locations (viewed from rear) and associated DA Pair numbering, for the High-Performance All-Flash configuration.


      Figure 5. Rear view DS8870 High-Performance All-Flash with 8 HPFEs installed

      The DS8870 Enterprise Class and the Business Class configuration support zero to eight HPFEs. Four installed in the base frame and 4 installed in the first expansion frame (Release 7.4 or later is required).
      Installation order is as follows; R1-B04, R1-B03, R1-B02, R1-B01, R2-B04, R2-B03, R2-B02, R2-B01,

      Figure 6 shows HPFE locations (viewed from rear) and associated DA Pair numbering for the Enterprise Class configuration. The Business Class configuration HPFE installation order and locations are exactly the same as the Enterprise Class.


      Figure 6. Rear view DS8870 Enterprise Class configuration with 8 HPFEs installed

      Table1 provides a summary of HPFE and Flash cards supported in DS8870 configurations. Business Class is not described, as the HPFE support is the same as Enterprise Class. For more information, see the IBM DS8870 Architecture and Implementation, SG24-8085 IBM Redbooks publication.

      Table 1. Summary of HPFEs support in DS8870 configurations


      Connectivity

      The High-Performance Flash Enclosures are directly attached to the I/O enclosure using a PCIe connectivity, which increases bandwidth and transaction-processing capability
      compared to standard drive enclosures connectivity (FC-AL). The flash interface cards extend the I/O enclosure PCIe connections to the HPFE. Each flash interface card is a PCIe
      redrive card that is located in the I/O enclosure.

      Figure 7 is a block diagram that shows a simplified view of the PCIe cabling topology.


      Figure 7. High-performance flash enclosure schematic cabling

      Figure 8 shows the locations for flash interface adapters in the DS8870 I/O enclosure, along with the host adapters (HAs) and standard enclosure device adapters (DAs).


      Figure 8. Flash interface adapters are shown in the I/O enclosure

      Easy Tier and flash cards

      IBM Easy Tier, a well-proven DS8870 feature is available at no charge. It dynamically optimizes performance for multi-tiered systems. It can also rebalance data within a single tier to help maintain optimal performance. Now in its sixth generation, Easy Tier offers full support for the high-performance flash enclosure, while maintaining additional features like Easy Tier Server, Easy Tier Application, and Easy Tier Heat Map Transfer.

      Note: All previously announced Easy Tier functions apply, including manual volume migration, automated sub-volume data relocation, automated performance rebalancing within drive tiers (in both single tier and multi-tier drive pools), hotspot management, and rank depopulation.

      Currently, IBM supports up to three drive classes or storage tiers that can be configured in the same DS8870:
      • Tier 0 or Flash tier: Contains flash cards and flash drives. Although flash cards are in the same tier as flash drives (SSDs), the Easy Tier intra-tier auto-rebalance function can exploit the higher IOPS capability of the HPFE.
      • Tier 1 or Enterprise tier: Contains the Enterprise drives (SAS 15 k or 10 k rpm).
      • Tier 2 or Nearline tier: Contains the Nearline drives (SAS 7.2 k rpm).

      For more information, see the IBM DS8870 Easy Tier, REDP-4667 IBM Redbooks publication.


      Disk Magic

      Disk Magic is a Windows based disk subsystem performance modeling tool that is used by IBM and IBM Business Partners to model disk storage subsystem performance. It supports disk systems from multiple vendors and offers detailed support for IBM storage systems. Contact your IBM Representative or IBM Business Partner to evaluate a Disk Magic study.

      Disk Magic supports the high-performance flash enclosure in the DS8870 with Licensed Machine Code (LMC) 7.7.30.xx.xx (bundle version 87.30.xx.xx) or later.


      Technical specifications summary

      Table 2 summarizes the high performance flash enclosure specifications.

      Table 2. HPFE technical specifications summary



      Upgrades

      The IBM DS8870 High-Performance Flash Enclosures are supported in the DS8870 with Licensed Machine Code (LMC) 7.7.3x.yy or later.
      HPFEs are supported in the first expansion frame of Enterprise Class and Business Class configurations with Licensed Machine Code (LMC) 7.7.4x.yy or later.

      With respect to HPFE upgrades within the DS8870:
      • Installing HPFE in DS8870 is non disruptive
      • HPFEs two to eight can be added to the DS8870 High-Performance All-Flash configuration (The first HPFE is required from manufacturing)
      • HPFEs one to four can be added to the base frame of the DS8870 Enterprise and Business Class configurations
      • HPFEs one to four can be added to the first expansion frame of the DS8870 Enterprise and Business Class configurations
      • Adding the second flash card set of 14 flash cards to any HPFE with 16 Flash Cards, to make the HPFE fully populated with 30 flash cards..
      Related publications and links

      For more information, see the following links:

      Special Notices

      This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment. publib-b.boulder.ibm.com

      Profile

      Publish Date
      20 May 2014

      Last Update
      20 November 2014


      Rating: Not yet rated


      Author(s)

      IBM Form Number
      TIPS1181