This IBM® Redbooks® product guide describes the DS8870 high-performance flash enclosure (HPFE).
Related Blog Posts
The high-performance flash enclosure (HPFE) is a Redundant Array of Independent Disks (RAID) storage enclosure that can support sixteen or thirty 400 GB encryption capable flash cards (1.8-inch, 46 mm form factor) in a 1U rack space.
HPFEs can be installed in the IBM DS8870 storage system. As of DS8870 microcode Release 7.4:
- One to eight HPFEs can be installed in the DS8870 High-Performance All-Flash configuration
- Up to four HPFEs can be installed in the DS8870 Enterprise Class and Business Class configuration base frame, and up to four HPFEs in the first expansion frame
Figure 1 shows the IBM® DS8870 high-performance flash enclosure.
Figure 1. Perspective view of HPFE
Did you know?
Compared to the flash drives (also known as solid-state drives or SSDs) that were already available for the DS8870, the high-performance flash enclosure provides a higher standard of flash performance. As implemented in the DS8870, the HPFE is directly attached to the PCIe fabric, enabling increased bandwidth compared to standard Fibre Channel-attached storage enclosures.
Flash cards are high-input/output operations per second (IOPS) class enterprise storage devices that are targeted at I/O-intensive workload applications that can benefit from a high level of fast-access storage.
High Performance Flash cards, along with flash drives, are considered as Tier 0 storage. Both offer a number of potential benefits over spinning drives, including better IOPS, lower power consumption, less heat generation, and lower acoustical noise. However, the flash cards offer even higher throughput using the flash RAID adapters in the high-performance flash enclosure and direct PCIe connectivity to the processor complexes, compared to the fiber attached flash drives installed in the standard drive enclosures.
The IBM Easy Tier® intra-tiering auto-rebalance feature is used to distribute the workload among traditional flash drives and flash cards according to their IOPS capacity within the highest storage tier (Tier 0).
High-performance flash enclosure highlights
The high-performance flash enclosure comes with various capabilities and performance characteristics, as characterized below:
- The High-Performance Flash Enclosure delivers up to 250,000 IOPS and up to 3.4 GBps bandwidth in just one EIA (1U) of rack space.
- Database performance can be accelerated by up to four times
- Batch times can shrink by up to 10%, therefore reducing impact on production applications.
- Each High-Performance Flash Enclosure contains up to 12 TB of raw capacity.
- Up to 8 HPFEs per DS8870, for a total of 96 TB raw capacity
- Flash cards in the High-Performance Flash Enclosure support encryption
Architecture and key components
The high-performance flash enclosure is a 1U enclosure that integrates a fully redundant pair of flash RAID adapters, dual power supplies with integrated cooling fans and sixteen or thirty 1.8-inch encryption capable flash cards.
The flash RAID adapters are directly connected over a PCIe fabric to the DS8870 I/O enclosures, with a Gen2 PCIe cable. Each connection provides a 2 GBps full duplex connection.
Effectively, the high-performance flash enclosure combines the RAID device adapter into the storage enclosure. The flash RAID adapter is a device adapter much like the device adapter in traditional Fibre Channel Arbitrated Loop (FC-AL) topology. The flash RAID adapter includes an SAS expander providing enclosure management services.
The graphic displayed in Figure 2 shows the major components of the HPFE. Top cover and cover from one of the Flash RAID adapters has been removed.
Figure 2. High Performance Flash Enclosure - Key components
Flash RAID adapter
Each HPFE contains two flash RAID adapters that provide redundant data paths to the flash cards in the enclosure. These adapters also provide enclosure services to control power, cooling, and other non-data functions. Each flash RAID adapter has there own redundant cooling fans.
Each flash RAID adapter connects to a DS8870 I/O enclosure pair by a 2 GB PCIe Gen 2 cable. The PCIe connectivity has none of the protocol overhead associated with Fibre Channel architecture. Each I/O enclosure pair supports up to two HPFEs.
Enclosure Power Supplies
Each HPFE has a pair of fully redundant power supply units (PSU). Each power supply unit has its own integrated fans. Cooling fans from the PSUs and the flash RAID adapters provide cooling for the entire enclosure.
The enclosure midplane provides the connectivity for the Flash RAID adapters, PSUs, and Flash cards.
High Performance Flash Cards
Each HPFE is installed with a minimum of sixteen 1.8 inch High Performance Flash Cards. An optional second set of 14 flash cards can be installed for a maximum total of thirty flash cards. If the second set is not installed, then a set of fillers would be installed in the empty slots.
The flash cards are installed into the HPFE in rows. See Figure 3 for Flash card locations, when HPFE is installed horizontally and vertically.
Figure 3. Flash card locations within the HPFE
The HPFE contains 16 or 30 flash cards. Implemented in the DS8870, HPFE only supports RAID 5 arrays. The HPFE will always have two spare flash cards.
The arrays are configured as follows;
- First 16 flash cards installed
- - arraySite 1 = 6+P+S RAID 5
- arraySite 2 = 6+P+S RAID 5
- Adding optional 14 flash card set installed
- - arraySite 3 = 6+P RAID 5
- arraySite 4 = 6+P RAID 5
High-Performance Flash Enclosures Implemented in the DS8870
The DS8870 has three configurations:
- The DS8870 High-Performance All-Flash configuration
- Supports one to eight HPFEs in a single frame
- Does not support any standard drive enclosures
- Offers maximum performance and connectivity in a single frame footprint
- The DS8870 Enterprise Class configuration
- Supports up to four HPFEs in the base frame
- Supports up to four HPFEs in the first expansion frame (Requires Release 7.4 or later)
- Supports up to 3 expansion frames
- Supports up to 1536 2.5 inch drives in standard drive enclosures
- Optimized for performance and is highly scalable, offering a wide range of options for long term growth.
- The DS8870 Business Class configuration
- Supports up to four HPFEs in the base frame
- Supports up to four HPFEs in the first expansion frame (Requires Release 7.4 or later)
- Supports up to 2 expansion frames
- Supports up to 1056 2.5 inch drives in standard drive enclosures
- Increases device adapter utilization, prioritizing cost-effective storage capacity growth.
Figure 4. High-Performance All-Flash and Enterprise Class configurations base frames with HPFEs
Population order and plug order
The DS8870 High-Performance All-Flash configuration supports a minimum of one HPFE installed in location R1-04.
Installation order is as follows; R1-B04 , R1-B03 , R1-B02, R1-B01, R1-B14, R1-B13, R1-B12, R1-B11
Figure 5 shows HPFE locations (viewed from rear) and associated DA Pair numbering, for the High-Performance All-Flash configuration.
Figure 5. Rear view DS8870 High-Performance All-Flash with 8 HPFEs installed
The DS8870 Enterprise Class and the Business Class configuration support zero to eight HPFEs. Four installed in the base frame and 4 installed in the first expansion frame (Release 7.4 or later is required).
Installation order is as follows; R1-B04, R1-B03, R1-B02, R1-B01, R2-B04, R2-B03, R2-B02, R2-B01,
Figure 6 shows HPFE locations (viewed from rear) and associated DA Pair numbering for the Enterprise Class configuration. The Business Class configuration HPFE installation order and locations are exactly the same as the Enterprise Class.
Figure 6. Rear view DS8870 Enterprise Class configuration with 8 HPFEs installed
Table1 provides a summary of HPFE and Flash cards supported in DS8870 configurations. Business Class is not described, as the HPFE support is the same as Enterprise Class. For more information, see the IBM DS8870 Architecture and Implementation, SG24-8085 IBM Redbooks publication.
Table 1. Summary of HPFEs support in DS8870 configurations
The High-Performance Flash Enclosures are directly attached to the I/O enclosure using a PCIe connectivity, which increases bandwidth and transaction-processing capability
compared to standard drive enclosures connectivity (FC-AL). The flash interface cards extend the I/O enclosure PCIe connections to the HPFE. Each flash interface card is a PCIe
redrive card that is located in the I/O enclosure.
Figure 7 is a block diagram that shows a simplified view of the PCIe cabling topology.
Figure 7. High-performance flash enclosure schematic cabling
Figure 8 shows the locations for flash interface adapters in the DS8870 I/O enclosure, along with the host adapters (HAs) and standard enclosure device adapters (DAs).
Figure 8. Flash interface adapters are shown in the I/O enclosure
Easy Tier and flash cards
IBM Easy Tier, a well-proven DS8870 feature is available at no charge. It dynamically optimizes performance for multi-tiered systems. It can also rebalance data within a single tier to help maintain optimal performance. Now in its sixth generation, Easy Tier offers full support for the high-performance flash enclosure, while maintaining additional features like Easy Tier Server, Easy Tier Application, and Easy Tier Heat Map Transfer.
Note: All previously announced Easy Tier functions apply, including manual volume migration, automated sub-volume data relocation, automated performance rebalancing within drive tiers (in both single tier and multi-tier drive pools), hotspot management, and rank depopulation.
Currently, IBM supports up to three drive classes or storage tiers that can be configured in the same DS8870:
- Tier 0 or Flash tier: Contains flash cards and flash drives. Although flash cards are in the same tier as flash drives (SSDs), the Easy Tier intra-tier auto-rebalance function can exploit the higher IOPS capability of the HPFE.
- Tier 1 or Enterprise tier: Contains the Enterprise drives (SAS 15 k or 10 k rpm).
- Tier 2 or Nearline tier: Contains the Nearline drives (SAS 7.2 k rpm).
For more information, see the IBM DS8870 Easy Tier, REDP-4667 IBM Redbooks publication.
Disk Magic is a Windows based disk subsystem performance modeling tool that is used by IBM and IBM Business Partners to model disk storage subsystem performance. It supports disk systems from multiple vendors and offers detailed support for IBM storage systems. Contact your IBM Representative or IBM Business Partner to evaluate a Disk Magic study.
Disk Magic supports the high-performance flash enclosure in the DS8870 with Licensed Machine Code (LMC) 7.7.30.xx.xx (bundle version 87.30.xx.xx) or later.
Technical specifications summary
Table 2 summarizes the high performance flash enclosure specifications.
Table 2. HPFE technical specifications summary
The IBM DS8870 High-Performance Flash Enclosures are supported in the DS8870 with Licensed Machine Code (LMC) 7.7.3x.yy or later.
HPFEs are supported in the first expansion frame of Enterprise Class and Business Class configurations with Licensed Machine Code (LMC) 7.7.4x.yy or later.
With respect to HPFE upgrades within the DS8870:
- Installing HPFE in DS8870 is non disruptive
- HPFEs two to eight can be added to the DS8870 High-Performance All-Flash configuration (The first HPFE is required from manufacturing)
- HPFEs one to four can be added to the base frame of the DS8870 Enterprise and Business Class configurations
- HPFEs one to four can be added to the first expansion frame of the DS8870 Enterprise and Business Class configurations
- Adding the second flash card set of 14 flash cards to any HPFE with 16 Flash Cards, to make the HPFE fully populated with 30 flash cards..
For more information, see the following links:
- DS8870 cross-reference table of code bundles
- DS8870 support
- IBM Disk Storage Feature Activation (DSFA)
- IBM Encrypted Storage Overview and Customer Requirements
- IBM publication IBM System Storage DS Command-Line Interface User’s Guide for DS8000 series, GC27-4212
- IBM publication IBM System Storage DS8000 242x Model 961 Installing a Storage Facility, GA32-2236
- IBM publication IBM System Storage DS8870 Introduction and Planning Guide, GC27-4209
- IBM Redbooks: IBM System Storage DS8000 Disk Encryption, REDP-4500
- IBM Redbooks: IBM System Storage DS8000 Easy Tier, REDP-4667:
- IBM Redbooks: IBM System Storage DS8000 Easy Tier Application, REDP-4514
- IBM Redbooks: IBM System Storage DS8000 Easy Tier Server, REDP-4513
- IBM Redbooks: IBM System Storage DS8000 Easy Tier Heat Map Transfer Server, REDP-4515
- IBM Redbooks: IBM System Storage DS8870 Architecture and Implementation, SG24-8085
- IBM DS8870 next-generation flash systems deliver high availability and better performance for critical environments - Product Announcement (May, 2014)
- IBM System Storage DS8000 Information Center
- IBM System Storage Interoperation Center (SSIC)
- Introduction to PCI Express
- Planned IBM Easy Tier integration with server-based SSDs (Video)
This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment.