This IBM® Redbooks® product guide describes the DS8870 high-performance flash enclosure (HPFE).
Related Blog Posts
The high-performance flash enclosure (HPFE) is a Redundant Array of Independent Disks (RAID) storage enclosure that can support up to thirty 400 GB encryption capable flash cards (1.8-inch, 46 mm form factor) in a 1U rack space. The high-performance flash enclosures and flash cards can only be installed in the DS8870 base frame.
The DS8870 Release 7.3 firmware can support the following configurations:
- The DS8870 Enterprise Class or Business Class configurations can support up to four HPFEs in a base frame, along with up to 10 standard drive enclosures.
- The DS8870 High Performance All Flash configuration can contain up eight HPFEs (four vertically and four horizontally) in a single base frame, for a maximum of 240 flash cards. No standard drive enclosures can be added.
Figure 1 shows the IBM® DS8870 high-performance flash enclosure.
Figure 1. Perspective view of HPFE
Did you know?
Compared to the flash drives (also known as solid-state drives or SSDs) that were already available for the DS8870, the high-performance flash enclosure provides a new and higher standard of flash performance. As implemented in a DS8870, the HPFE is directly attached to the PCIe fabric, enabling increased bandwidth compared to standard Fibre Channel-attached storage enclosures.
Flash cards are high-input/output operations per second (IOPS) class enterprise storage devices that are targeted at I/O-intensive workload applications that can benefit from a high level of fast-access storage.
Flash cards along with flash drives are considered as part of Tier 0. Both types offer a number of potential benefits over spinning drives, including better IOPS, lower power consumption, less heat generation, and lower acoustical noise. However, the flash cards offer even higher throughput using the flash RAID adapters in the high-performance flash enclosure and direct PCIe connections to the processor complexes.
The IBM Easy Tier® intra-tiering auto-rebalance (micro-tiering) feature is used to distribute the workload among traditional flash drives and flash cards according to their IOPS capacity within the highest storage tier (Tier 0).
High-performance flash enclosure highlights
The high-performance flash enclosure comes with various capabilities and performance characteristics, as summarized here:
- The high-performance flash enclosure delivers up to 250,000 IOPS and up to 3.4 GBps bandwidth in just one EIA (1U) of rack space.
- Using high-performance flash enclosures can accelerate database performance by up to four times while still guaranteeing the industry-leading DS8870 availability.
- By using high-performance flash enclosures, batch times shrink by up to 10%, therefore reducing impact on production applications.
- High-performance flash enclosures can carry up to 12 TB of raw capacity per enclosure.
- Flash cards in the high-performance flash enclosure support full drive encryption.
Architecture and key components
The high-performance flash enclosure is a 1U enclosure that integrates a fully redundant pair of flash RAID adapters, dual power supplies with integrated cooling fans and up to thirty 1.8-inch encryption capable flash cards.
The flash RAID adapters are directly connected over a PCIe fabric to the DS8870 I/O enclosures, with a Gen2 PCIe cable. Each connection provides a 2 GBps full duplex connection.
Effectively, the high-performance flash enclosure combines the RAID device adapter into the storage enclosure. The flash RAID adapter is a device adapter much like the device adapter in traditional Fibre Channel Arbitrated Loop (FC-AL) topology. The flash RAID adapter includes an SAS expander providing enclosure management services.
Each high-performance flash enclosure contains the following hardware components:
- 16 or 30 400-GB 1.8-inch SAS flash cards, which support IBM Full Disk Encryption (FDE).
- Figure 2 shows the 1.8-inch SAS flash cards.
Figure 2. 1.8-inch flash cards assembly
- If the high-performance flash enclosure is ordered with only 16 flash cards, 14 flash card fillers are installed in the empty flash card locations. When 14 additional flash cards are installed, to fully populate the enclosure, the fillers are removed.
- Two power supplies with integrated cooling fans, as shown in Figure 4.
Figure 4. Two HPFE power supply units
- Each HPFE contains two flash RAID adapters that provide redundant data paths to the flash cards in the flash enclosure. These adapters also provide enclosure services to control power, cooling, and other non-data functions.
Note: Flash cards are not available for capacity on demand (CoD) configurations.
The flash cards are installed into the HPFE in rows, starting from its upper left side (in case of horizontal installation). See Figure 3.
Figure 3. Front view of high-performance flash enclosure in horizontal positions
- The dual integrated RAID controllers in the DS8870 high-performance flash enclosures are connected to the I/O Enclosures via PCIe cable. Each cable supports a 2 GB bandwidth (bidirectional), with none of the protocol overhead associated with Fibre Channel architecture. Each I/O enclosure pair supports up to two HPFEs. Figure 5 shows the rear view of the HPFE.
Figure 5. Rear view of the HPFE
- One back plane for plugging components. Figure 6 shows where the components are located.
Figure 6. Back plane assembly
The high-performance flash enclosures can each contain up to four array sites. The first group of 16 Flash Cards (required) in each flash enclosure will form two 8-flash card array sites. Each of these two array sites will designate one flash card as a spare card. The second set of 14 cards (optional) will form two 7-card array sites and no other spare will be created, for a total of two spares per flash enclosure. This configuration is unique to the high-performance flash enclosure. This is a change from the array logical configuration process used with standard drive enclosures, where all array sites have eight drives.
During the configuration process, arrays are created from the array sites. The flash enclosures support only RAID 5 arrays, which will be either 6+P+S or 6+P.
This section gives an overview of the possible configurations for DS8870 including the high-performance flash enclosure.
Figure 7 shows a table with all configurations possible for DS8870, highlighting the maximum number of flash cards, flash drives, and spinning disks.
Figure 7. Table showing DS8870 possible configurations
The high-performance flash enclosure is available in the following configurations, which are depicted in Figure 8:
- DS8870 Model 961 Enterprise Class or Business Class configuration:
- In the Enterprise or Business Class configuration, you can have up to four HPFEs installed vertically in the base frame. Each HPFE can have either 16 or 30 flash cards. The base frame can still accommodate up to 10 standard drive enclosures, populated with either flash drives (SSDs) or spinning drives.
- DS8870 Model 961 High Performance All Flash configuration
- In the High Performance All Flash configuration, up to 8 HPFEs (four horizontally and four vertically) can be installed in the base frame. A High Performance All Flash configuration has four I/O enclosure pairs installed expanding the number of hosts and maximizing the bandwidth with up to 16 four or eight port Fibre Channel/IBM FICON® host adapters, offering maximum performance and connectivity in a single frame footprint.
- The High Performance All Flash configuration does not support any standard disk enclosures or expansion frames.
Figure 8. DS8870 base frame with HPFEs
The high-performance flash enclosures are directly attached to the I/O enclosure by using a PCIe connection, which increases bandwidth and transaction-processing capability compared to standard drive enclosures connectivity (FC-AL). The flash interface cards extend the I/O enclosure PCIe connections to the flash enclosure. Each flash interface card is a PCIe redriver card that plugs into the I/O enclosure.
Figure 9 is a block diagram that shows a simplified view of the PCIe cabling topology.
Figure 9. High-performance flash enclosure schematic cabling
Figure 10 shows the locations for flash interface adapters in the DS8870 I/O enclosure, along with the host adapters (HAs) and standard enclosure device adapters (DAs).
Figure 10. Flash interface adapters are shown in the I/O enclosure
Population order and plug order
This section explains the plug order of HPFEs in the DS8870 961 frame, and applies whether it is an All Flash, Business Class, or Enterprise Class configuration.
The DA pairs to which the HPFEs are attached are numbered 8 - 15. In Business or Enterprise Class configuration, HPFEs attaching to DA pairs 8 - 11 are installed vertically in the following sequence, 10->8->11->9, while in High Performance All Flash configurations, HPFEs attaching to DA pairs 8 - 15 are installed in the sequence 10->8->11->9->14->12->15->13.
Each HPFE is connected by its two flash RAID adapters to two different Flash Interface Cards (FICs), each of them in a different I/O bay.
Figure 11 (for Business Class and Enterprise Class) and Figure 12 (for All Flash) show HPFE connections and plug order with colors and numbers that indicate the corresponding connected Flash RAID adapters and FICs.
Figure 11. Enterprise class HPFEs plug order (only base frame shown)
Figure 12. All Flash system HPFEs plug order
Easy Tier and flash cards
IBM Easy Tier, a well-proven DS8870 feature is available at no charge. It dynamically optimizes performance for multi-tiered systems. It can also rebalance data within a single tier to help maintain optimal performance. Now in its sixth generation, Easy Tier offers full support for the high-performance flash enclosure, while maintaining additional features like Easy Tier Server, Easy Tier Application, and Easy Tier Heat Map Transfer.
Note: All previously announced Easy Tier functions apply, including manual volume migration, automated sub-volume data relocation, automated performance rebalancing within drive tiers (in both single tier and multi-tier drive pools), hotspot management, and rank depopulation.
Currently, IBM supports up to three drive classes or storage tiers that can be configured in the same DS8870:
- Tier 0 or Flash tier: Contains flash cards and flash drives. Although flash cards are in the same tier as flash drives (SSDs), the Easy Tier intra-tier auto-rebalance function can exploit the higher IOPS capability of the HPFE.
- Tier 1 or Enterprise tier: Contains the Enterprise drives (SAS 15 k or 10 k rpm).
- Tier 2 or Nearline tier: Contains the Nearline drives (SAS 7.2 k rpm).
For more information, see the IBM DS8870 Easy Tier, REDP-4667 IBM Redbooks publication.
Disk Magic is a Windows based disk subsystem performance modeling tool that is used by IBM and IBM Business Partners to model disk storage subsystem performance. It supports disk systems from multiple vendors and offers detailed support for IBM storage systems. Contact your IBM Representative or IBM Business Partner to evaluate a Disk Magic study.
Disk Magic supports the high-performance flash enclosure in the DS8870 with Licensed Machine Code (LMC) 7.7.30.xx.xx (bundle version 87.30.xx.xx) or later.
Technical specifications summary
Table 1 summarizes the high performance flash enclosure specifications.
Table 1. DS8870 high-performance flash enclosure specifications
|Maximum Quantity in a system||8 pcs (4 pcs in Business or Enterprise Classes)|
|Flash Card interface||6 Gbps SAS Flash Drive|
|Number of disk drives (minimum/maximum)||16/30|
|Maximum physical storage capacity||12 TB (9.11 TB usable capacity, RAID 5)|
|Disk sizes||400 GB|
For more information, see the IBM DS8870 Introduction and Planning Guide, GC27-4209 IBM publication.
The IBM DS8870 high-performance flash enclosures are only supported in the DS8870 with Licensed Machine Code (LMC) 7.7.30.xx.xx.
Conversion of Business class or Enterprise configuration into a High Performance All Flash configuration is not supported.
Related publications and links
For more information, see the following links:
- DS8870 cross-reference table of code bundles
- DS8870 support
- IBM Disk Storage Feature Activation (DSFA)
- IBM Encrypted Storage Overview and Customer Requirements
- IBM publication IBM System Storage DS Command-Line Interface User’s Guide for DS8000 series, GC27-4212
- IBM publication IBM System Storage DS8000 242x Model 961 Installing a Storage Facility, GA32-2236
- IBM publication IBM System Storage DS8870 Introduction and Planning Guide, GC27-4209
- IBM Redbooks: IBM System Storage DS8000 Disk Encryption, REDP-4500
- IBM Redbooks: IBM System Storage DS8000 Easy Tier, REDP-4667:
- IBM Redbooks: IBM System Storage DS8000 Easy Tier Application, REDP-4514
- IBM Redbooks: IBM System Storage DS8000 Easy Tier Server, REDP-4513
- IBM Redbooks: IBM System Storage DS8000 Easy Tier Heat Map Transfer Server, REDP-4515
- IBM Redbooks: IBM System Storage DS8870 Architecture and Implementation, SG24-8085
- IBM DS8870 next-generation flash systems deliver high availability and better performance for critical environments - Product Announcement (May, 2014)
- IBM System Storage DS8000 Information Center
- IBM System Storage Interoperation Center (SSIC)
- Introduction to PCI Express
- Planned IBM Easy Tier integration with server-based SSDs (Video)
This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment. publib-b.boulder.ibm.com