Abstract
This IBM Redbooks publication provides a comprehensive introduction to the IBM Storage Scale System family of integrated hardware and software storage solutions. Built on IBM Storage Scale technology, the IBM Storage Scale System delivers high-performance, reliable file and object storage designed for AI, high-performance computing (HPC), analytics, and hybrid cloud workloads.
The publication covers the IBM Storage Scale System 6000 and 3500 product lines, describing their architecture, hardware components, software stack, and deployment models. It explains how IBM Storage Scale System functions as a scalable storage building block within an IBM Storage Scale cluster, supporting massive scalability, software-defined RAID with de-clustered erasure coding, and flexible network connectivity including 100/200/400 Gb Ethernet and InfiniBand HDR/NDR.
Authors
Phillip Gerrard, Sagrika Arora, Nilton Carlos Dos-Santos, Jordi Caubet, Michael Kieran, Stieg Klein, Chris Maestas and Bis Purkayastha
- Introduction
- Introducing the IBM Storage Scale System
- IBM Storage Scale System architecture
- Architecture overview
- IBM Storage Scale System hardware components
- Software components
- IBM Storage Scale System planning and integration
- IBM Storage Scale System models
- Abbreviations
- Additional Resources
- Notices
- Version History
IBM Storage Scale System architecture
This section describes the architecture, hardware, and software of the IBM Storage Scale System, including the following components:
- IBM Storage Scale Management Server (IBM EMS)
- IBM Storage Scale Utility Node (Services for Management, Protocol, Active file Management (AFM), Protect and in the future Data Cataloging)
- I/O and data servers
- Storage enclosures
This section also describes the following building block models, software components, and features:
- Operating system
- Ansible playbooks and container-based software upgrades
- IBM Storage Scale
- IBM Storage Scale RAID
Note: IBM Storage Scale System is an integrated packaged solution. Users should not install different kernel levels or drivers into the IBM Storage Scale System. The user should also not run any non-IBM Storage Scale System (client) application or non IBM Storage Scale System workloads on the I/O nodes or IBM Storage Scale System Utility Node Servers.
Architecture overview
An IBM Storage Scale System is a fully integrated solution that combines hardware and software, designed and tested to operate as a single, unified system. The major components of an IBM Storage Scale System solution release include the following items:
- Server hardware
- Storage hardware
- IBM Storage Scale software including IBM Storage Scale RAID
- Embedded Red Hat Enterprise Linux (RHEL) operating system
The minor components, which are integration-tested with the major components, include (but not limited to) the following items:
- Server firmware
- Server host bus adapters (HBAs) and related firmware
- Nvidia network adapter, drivers, firmware, and network switches
- Storage enclosure and drive firmware
- IBM racks, power distribution units (PDUs), and cabling management
In addition, many of the hardware and software components of the IBM Storage Scale System solution communicate with each other over an IP network. IBM Storage Scale System requires all networking connections to be in place and provisioned to install the system. This IP network can use Nvidia network switches, which are tested with IBM Storage Scale System by IBM, or the network switches can be provided by the client. These IP network components consist of the following switches:
- Low-speed 1 Gb and 10 Gb Ethernet (GbE) network switches for management and service networks
- High-speed network switches for data read and write over a high-speed data network
IBM Storage Scale System hardware components
This section includes a description of the following hardware components:
- Solution models:
- 6000
- 3500
- Server hardware
Current generation IBM Storage Scale System 6000 and 3500 use x86 based servers for storage.
- Storage enclosures
- Network interface cards (NICs) and network switches
- Rack
IBM Storage Scale Systems are a rack-mounted storage solution. The initial storage building block in an IBM Storage Scale System cluster includes the following components:
- One IBM Storage Scale Management Server: Required for every IBM Storage Scale cluster
- One or more IBM Storage Scale System, each of which includes redundant I/O data servers
Current generation IBM Storage Scale System models include various models that provide NVMe, serial-attached SCSI (SAS), or Serial Advanced Technology Attachment (SATA) hard disk drives (HDDs) in selected sizes.
The IBM Storage Scale System 6000 was announced and delivered in 4Q2023. The IBM Storage Scale System 6000 design builds on the IBM Storage Scale System 3500, with more powerful processors and additional PCIe Gen 5 x 16 lanes that are both faster and wider than those supported by the 3500. Additionally, several design updates improve serviceability and provide additional system configuration flexibility, such as an optional campus connection.
The IBM Storage Scale System 3500 was announced and delivered in 2Q2022. The IBM Storage Scale System 3500 design is based on prior IBM Storage Scale System models. The IBM Storage Scale System 3500C/H models support up to eight external storage enclosures.
Available IBM Storage Scale System models
At the time of writing, two IBM Storage Scale System models are available to order from IBM.
IBM Storage Scale System 6000
The IBM Storage Scale System 6000 machine type 5149-F48 for the 4U48 form factor includes the following features:
- Embedded NVMe TLC drive options with a choice of 3.84 TB, 7.68 TB, 15.36 TB, or 30.72 TB TLC Gen5 drives.
- Embedded mixed NVMe TLC + QLC drive options with a choice of 30.72 TB, 61.44 TB, or 122.8 TB QLC drives.
- Embedded NVMe options IBM FlashCore® Modules with 38.4 TB drives. IBM FlashCore Modules have inline compression enabled.
- Optionally extended with HDD expansion enclosures (MTM 5149-091) supporting 91 SAS HDDs per enclosure with a choice of 12TB, 16 TB, 20 TB, 22 TB, or 24 TB Self-Encrypting Drives (SED) and up to 9 expansion enclosures per system.
- Optionally extended with all-flash expansion enclosures supporting 26 NVMe QLC drives per enclosure with a choice of 30.72 TB, 61.44 TB or 122.8 TB drives and up to 4 expansion enclosures per system.
IBM Storage Scale System 3500
The IBM Storage Scale System 3500 machine type 5141-FN2 for the 2U24 form factor includes the following features:
- Embedded NVMe TLC drive options with a choice of 3.84 TB, 7.68 TB, 15.36 TB, or 30.72 TB TLC Gen5 drives.
- Optionally extended with up to 8 HDD expansion enclosures (MTM 5147-102). Each supports 102 SAS HDDs per enclosure with a choice of 10 TB, 14 TB, 18 TB, 20 TB, 22 TB, or 24 TB, with support for Self-Encrypting Drives (SED) capability.
For more information about previous IBM Storage Scale System models, see the IBM Redbooks publication
IBM Storage Scale System 6000
IBM Storage Scale System 6000 is an all-flash, integrated, end-to-end NVMe storage solution, providing industry-leading performance, scalability, and unrivaled data protection and availability. The 6000 provides twice the performance and storage capacity relative to the previous generation systems and is the most sustainable storage solution to date.
IBM Storage Scale System 6000 is a high performance, accelerated storage offering optimized for data intensive workloads for advanced AI, analytics, and high-performance computing. The 6000 delivers significant advancements to the data services provided by the global data platform.
The Storage Scale System 6000 uses the power of Storage Scale combined with NVMe and flash technology in an x86 PCIe Gen5 4U48 form factor to deliver high-performance storage for AI, data analytics, and any high-performance file and object use cases. With support for IBM Flash Core Module technology, the 6000 supports up to 1.8 PB of raw flash capacity and
5.5 PB of compressed flash storage in a single 4U building block, as shown in the following figure.
The Storage Scale System 6000 can deliver more than 300 gigabytes per second (GBps) of performance with low latency to optimize all types of I/O patterns and data layouts. This performance delivers data to applications faster and helps ensure maximum GPU resource usage. Performance can vary because of disk technology, high-speed network adapters, workload, and configuration. To estimate performance for your environment, contact an IBM representative or IBM Business Partner.
The Storage Scale System 6000 supports up to 16 ports of 100 Gb RDMA over Converged Ethernet (RoCE), InfiniBand HDR 200, InfiniBand NDR 200/400, or a combination of these.
The two redundant high-performance controllers in a 2U platform provide high availability and reliability. An embedded baseboard management controller (BMC) with remote control capability allows administrators to manage the system remotely.
IBM Storage Scale System is an accelerated storage offering optimized for data intensive workloads for advanced AI, analytics, and high-performance computing.
IBM Storage Scale System 6000 can be an ideal storage platform for the following applications:
- Compute intensive AI model training and inferencing for GPU-accelerated workloads requiring low latency, and high OPS
- Accelerated AI, delivering performance, scalability, data virtualization, and data resiliency for IBM watsonx.data® and IBM watsonx.ai®
- Industry-leading1 throughput for data intensive high-performance computing solutions
IBM Storage Scale System is designed so that you can break data access barriers and create new storage dynamics for Nvidia AI solutions. With IBM Storage for data and AI, you can create a global data platform with top-rated performance to accelerate data access greater than 125 GB/s per node with parallel data access for GPU-accelerated computing with NVIDIA DGX systems.
IBM Storage Scale System 6000 Performance Model
IBM Storage Scale System 6000 Performance Model refers to the machine type 5149-F48 with no external enclosures. IBM Storage Scale System 6000 control enclosure provides 48x NVMe slots at the front redundantly accessible by the two I/O servers at the back of the enclosure.
A system can be configured with 24 drives in a half-populated configuration or 48 drives in a fully populated configuration. Two NVMe drive options can be configured:
- NVMe TLC drives with a choice of 3.84 TB, 7.68 TB, 15.36 TB, or 30.72 TB TLC Gen5 drives.
- IBM FlashCore® Modules with a choice of 38.4 TB drives. IBM FlashCore Modules have inline compression enabled.
IBM Storage Scale System 7.0.1 release introduces a 12-drives quarter-populated configuration providing a small starter configuration.
The following table provides an overview of the IBM Storage Scale System 6000 server canister specifications.
- Although one third and two third memory configurations are possible, number of drives - type and size - and number of enclosures will force a minimum memory footprint
IBM Storage Scale System version 6.2.3 introduces the capability to mix TLC drives and QLC drives. A system can be configured with 24 TLC drives and 24 QLC drives with a tiered flash storage configuration in the same enclosure. IBM Storage Scale System 7.0.0 release includes support so that you can configure 12x TLC drives and 36x QLC drives. All drives on each tier must have the same capacity. QLC drive options include 30.72 TB, 61.44 TB, or 122.8 TB. Support for the 122.8 TB drive was introduced with version 7.0.0.1
By default, IBM Storage Scale System 6000 Performance Model can configure up to four adapter slots for use with high-speed network cards. The four adapter slots that are used are located in the middle of the device, as shown in the following figure. The other adapter slots are kept empty for future MES to connect external storage expansion enclosures.
IBM Storage Scale System 6000 with HDD storage expansion enclosures (MTM 5149-091)
IBM Storage Scale System 6000 (machine type 5149-F48) can include external HDD storage expansion enclosures (5149-091). A system that has internal NVMe drives plus storage expansion enclosures is a Hybrid Model. The number of internal NVMe drives can be 24 drives for half populated or 48 drives for fully populated. The system that has only storage expansion enclosures is a Capacity Model and has no internal drives for data storage. The Capacity Model includes internal data drives that are used exclusively for log tip and other internal IBM Storage Scale usage but are not used to store data.
At the time of writing, you can add storage expansion enclosures in only nonmixed NVMe configurations. For instance, you can add them in combination with NVMe TLC or IBM FlashCore modules but not in combination with mixed TLC+QLC configurations.
A system can include up to 9x external expansion storage enclosures. Each enclosure includes 91 SAS HDDs with a choice of 12 TB, 16 TB, 20 TB, 22 TB, or 24 TB. All enclosure drive sizes in all enclosures must be the same. Using the MES process, you can add enclosures that are fully populated with the same existing drive size.
External HDD storage enclosures are connected by using SAS adapters that are installed on the four available PCI slots that are not used for high-speed network adapters. You can use the four SAS adapters in each server to connect up to 8 enclosures. To connect a ninth enclosure, six SAS adapters are required, which leaves room for only two high-speed network adapters.
IBM Storage Scale System 6000 with JBOF storage expansion enclosures (MTM 5149-F26)
IBM Storage Scale System version 7.0.0 introduces support for external Flash storage expansion enclosures (5149-F26). You can add these enclosures to only NVMe TLC-only configurations. Systems with IBM FlashCore modules or with mixed NVMe TLC and QLC drives do not support external Flash storage expansion enclosures.
Each 5149-F26 enclosure includes 26 NVMe QLC drives in one of three capacities: 30.72 TB, 61.44 TB, or 122.8 TB. Each enclosure ships fully populated with drives. You can attach up to four 5149-F26 enclosures per system. A system can use either HDD or JBOF external enclosures, but not both types on the same system.
External JBOF storage enclosures are connected by using NVMe over Fabrics (NVMe-oF) in direct-attached mode to the I/O canisters. In the four available PCI slots that are not used for high-speed network adapters, the system installs high-speed network adapter cards for such direct-attached connectivity instead of SAS adapters when you use HDD enclosures. These adapters, even if they are the same adapters used in high-speed adapter slots, are used only for backend connectivity, meaning direct attachment of I/O canisters to JBOF enclosures.
An existing Performance Model with TLC drives (24 or 48) can add 1, 2, 3, or 4 JBOF enclosures or it can be configured as an initial system. At the time of writing this Redbooks publication, the MES process to increase JBOF enclosures is not supported. However, always look to the latest documentation for further updates.
IBM Storage Scale System 3500
IBM Storage Scale System 3500 is an all-flash, integrated, end-to-end NVMe storage solution that provides industry-leading performance, scalability, data protection, and availability.
The Storage Scale System 3500 can provide over 125 gigabytes per second (GBps) of performance with low latency across all types of I/O patterns and data layouts. This delivers data to applications faster and helps ensure maximum GPU resource usage. Performance varies with disk technology and high-speed network adapters. For more information about estimating and sizing your performance, contact your IBM representative or IBM Business Partner representative.
The Storage Scale System 3500 supports up to 8 ports of 100 Gb RDMA over Converged Ethernet (RoCE), InfiniBand HDR 200, InfiniBand NDR 200, or a combination of both. Two redundant high-performance controllers in a 2U platform support high availability and reliability.
An embedded baseboard management controller (BMC) provides remote management capability.
IBM Storage Scale System is designed help you create new storage dynamics for Nvidia AI solutions. With IBM Storage for data and AI, you can create a global data platform with data access rates greater than 125 GB/s per node. By using the Storage Scale System 3500, you can achieve these speeds by using parallel data access for GPU-accelerated computing with high-performance, local NVMe storage and SAS-connected capacity storage.
IBM Storage Scale System 3500 can include 12 or 24 NVMe drives of 3.84 TB, 7.68 TB, 15.36 TB, or 30.74 TB for user data. For more information, see IBM Storage Scale System 3500.
IBM Storage Scale 3500 Capacity and Hybrid models can include 1–8 storage shelves (MTM 5147-102) of NL-SAS drives. When only one storage shelf is used, it can contain 52 or 102 drives. After expanding the first shelf to 102 drives, extra shelves that contain 102 drives can be added. Supported drive sizes are 10 TB, 14 TB, 18 TB, 20 TB, 22 TB, or 24 TB. For more information, see IBM Storage Scale System 5147-102 Storage Enclosure.
IBM Storage Scale System 3500 supports nondisruptive capacity upgrades including the addition of up to a maximum of 8 storage shelves. For each shelf, all HDDs must be of equivalent size and performance.
IBM Storage Scale System 3500 Performance Model
IBM Storage Scale System 3500 Performance Model refers to the machine type 5141-FN2 with no external enclosures. The IBM Storage Scale System 3500 control enclosure provides 24 NVMe slots that are redundantly accessible by the two I/O servers at the back of the enclosure.
A system can be configured with 12 drives in a half-populated configuration or 24 drives in a fully populated configuration. The supported NVMe drives are 3.84 TB, 7.68 TB, 15.36 TB, or 30.72 TB TLC Gen5 drives. On the Storage Scale System 3500, Gen5 drives operate as Gen4 drives because the internal PCI bus is Gen4.
IBM Storage Scale System 3500 with HDD storage expansion enclosures (MTM 5147-102)
IBM Storage Scale System 3500 (machine type 5141-FN2) can include external HDD storage expansion enclosures (5147-102). A system that has 12 or 24 NVMes drive and storage expansion enclosures is a Hybrid Model. The system that has only storage expansion enclosures is a Capacity Model. The Capacity Model includes four NVMe drives that are used exclusively for log tip and other internal IBM Storage Scale usage.
A system can include up to 8 external expansion storage enclosures, as shown in the preceding figures. Each enclosure includes 102 SAS HDDs with a choice of 10 TB, 14 TB, 18 TB, 20 TB, 22 TB and 24 TB. For each shelf or enclosure, all drives must have the same capacity and speed. By using the MES process, it is possible to add enclosures that are fully populated with the same drive size.
External HDD storage enclosures are connected by using SAS adapters that are installed in the two available PCI slots that are not used for high-speed network adapters. Two SAS adapters in each server provide the connection to up to 4 enclosures. You can connect a maximum of 8 enclosures that are connected by using daisy chain connections. After a 4th enclosure is added, any additional enclosures add only capacity, not additional performance.
Positioning various IBM Storage Scale System models
IBM Storage Scale System models are available in different categories. The x in the models denotes the number of storage enclosures in that model.
Consider the following points:
- IBM Storage Scale System 6000 models are built on NVMe flash storage for high performance. Each IBM Storage Scale System 6000 canister supports up to eight PCIe5 cards. PCI cards are able to be used for either networking or HBA attachment for external storage expansion. A single NIC supports up to two high-speed networking interfaces. A single HBA supports up to two high-capacity external storage enclosures.
- IBM Storage Scale System 3500 models are built on NVMe flash storage for high performance, and optionally high capacity 4U102 HDD storage enclosures. Each IBM Storage Scale System 3500 canister supports up to four PCIe4 cards. A single card either supports up to two high-speed networking interfaces or up to two high-capacity storage enclosures. IBM Storage Scale System components also require the following items:
- An IBM Storage Scale System Management Server
- An IBM Storage Scale System Protocol Node
IBM Storage Scale System 6000 models
For the following table, all flash media are SED and you cannot inter-mix NVMe and FCM4 in the same chassis.
IBM Storage Scale System 3500 models
For the following table, all flash media are SED.
IBM Storage Scale System usable capacity is a percentage of the raw capacity. The usable capacity varies, depending on the IBM Storage Scale RAID parity that is used. The default 8+2P parity is the most common configuration. Approximately 73% of the raw capacity is the usable capacity when you use the default 8+2P parity in external HDD storage enclosures. Small drive configurations provide a lower usable percentage.
IBM or IBM Business Partners can provide the exact usable capacities of the various models on request by using the IBM Storage Modeler (StorM) to calculate the exact usable capacity. Contact your IBM representative or IBM Business Partner representative for assistance in determining the usable capacity in your IBM Storage Scale System model.
Enclosures are fully redundant. For configurations that require enclosure-level protection, use the following erasure code options:
- 8+3P provides enclosure protection with four or more external enclosures.
- 8+2P provides enclosure protection with five or more external enclosures.
- Configurations with one, two, or three enclosures require three-way or four-way erasure code for enclosure protection.
IBM Storage Scale System 7.0.1 release provides new longer erasure codes, 16+2P and 16+3P, that provide better usable capacity.
For more information about the various IBM Storage Scale System models specifications, see IBM Storage Scale System Models.
IBM Storage Scale System Utility Node and Protocol Node
Released as part of IBM Storage Scale System 6.1.8.1, the IBM Storage Scale System Utility Node (MTM 5149-23E) replaced the existing IBM Power9® based platforms, both EMS and Protocol Nodes.
IBM Storage Scale System 3500 can be managed by an IBM Power9 EMS or an IBM Storage Scale System Utility Node. However, the IBM Storage Scale System 6000, released with IBM Storage Scale System version 6.1.9.1, can be managed by only the Utility Node. When you combine IBM Storage Scale System 3500 and 6000 on the same cluster, use the Utility Node instead of the legacy IBM Power9 EMS. Use the Utility Node to help ensure a unified management approach and to avoid multiple management points.
The following figures show the front and back of the Utility Node.
The same MTM 5149-23E is used for both the Management Server and Protocol Node, but it differs on hardware configuration and adapter placement rules.
The Utility Node was released as a Management Server, so it includes a single socket with only two adapter slots available: one slot for mandatory 4-port management card and one slot for a network adapter.
The release of IBM Storage Scale System version 6.2.3 introduces the option of a dual-socket Utility Node as a Management Server. The dual-socket Utility Node also includes an option for more memory and includes four adapter slots. The four adapter slots consist of a mandatory 4-port management card (always in slot 5) and three high-speed network adapter slots ( and ).
Utility Node for Protocol Node is a dual-socket server with different adapter rules than the Utility Node EMS dual-socket. For example, 4-port 10GbE is not needed on the Protocol Node so slot C5 is used for a high speed network adapter (4).
IBM Storage Scale System 6.2.3 release also introduces the capability to perform an MES of existing Utility Nodes. For single-socket or dual-socket configurations, you can change or update Utility Node memory. For dual-socket versions only, you can change or update Utility Node adapters.
With IBM Storage Scale System version 7.0.0, you can deploy the Utility Node protocol node by using the
With IBM Storage Scale System version 7.0.1, a 4-port 64 Gb Fibre Channel HBA is introduced as part of the supported and configurable adapters. Slot placement rules apply; see the IBM Storage Scale System Utility Node documentation.
Nvidia DGX SuperPod
IBM Storage Scale System is an approved storage solution for Nvidia DGX SuperPOD.
The Nvidia DGX SuperPOD with Nvidia DGX H200, Nvidia DGX B200 or Nvidia DGX B300 systems is an artificial intelligence (AI) supercomputing infrastructure. It provides the computational power necessary to train today's state-of-the-art deep learning (DL) models and to fuel future innovation. The DGX SuperPOD delivers groundbreaking performance, deploys as a fully integrated system, and is designed to solve the world's most challenging computational problems.
This DGX SuperPOD storage reference architecture (RA) is the result of collaboration between DL scientists, application performance engineers, and system architects to build a system capable of supporting the widest range of DL workloads. The performance that is delivered by the DGX SuperPOD with DGX systems enables the rapid training of DL models at great scale. The integrated approach of provisioning, management, compute, networking, and fast storage enables a diverse, multi-tenant system that can span data analytics, model development, and AI inference.
IBM Storage Scale is a parallel file system, and its optimal performance is achieved when multiple processes or workloads are running in parallel. To maximize the results of a high-performance computing (HPC) and AI cluster solution with IBM Storage Scale 6000 and Nvidia DGX SuperPOD architecture, see IBM Storage and Nvidia DGX SuperPOD with IBM Storage Scale and IBM Storage Scale System publications.
Networking switches
The IBM Storage Scale System solution requires IP or InfiniBand networking switches to interconnect the IBM Storage Scale System hardware components.
As a best practice, purchase and use Nvidia networking switches. These network switches are tested by IBM to work with IBM Storage Scale System.
Clients can also provide and support their own Ethernet or InfiniBand networking switches.
Types of networks that are used by IBM Storage Scale System
The IBM Storage Scale System features the following types of networks that are used to interconnect the IBM Storage Scale System components:
-
Management and service networks, which must have separate 1 Gb or 10 GbE networks for management and service traffic
-
Data network, which is a high-speed network for the data that is written into and read out of the IBM Storage Scale IBM Storage Scale System file systems
Regardless of the source and type of network switches between the IBM Storage Scale System, the IBM Storage Scale cluster and the users, the networking must be solid, robust, reliable, and able to provide consistent low-latency responses. Nonblocking, high-speed switches are highly recommended.
Experience shows that IBM Storage Scale and IBM Storage Scale System can easily generate throughput rates that can overwhelm an over-subscribed network. IBM Storage Scale provides a network load generation testing tool (
For more information about the
Management network switches
IBM Storage Scale System components require a low-speed Ethernet connection for scale management and separate baseboard management controller (BMC) traffic. As part of an IBM Storage Scale System order, IBM can supply the low-speed management network switches for IBM MTM 5149-S54.
As a best practice, order and use this 1/10 GbE network management switch because it is integrated into the IBM Storage Scale System solution at the manufacturing stage. This integration allows the entire IBM Storage Scale System to be installed and initially tested without external network dependencies.
Alternatively, the client can provide a 1/10 GbE network switch management infrastructure. In this case, the client is responsible for providing the suitable management IP networking port counts and network switch configuration and definitions, including VLANs.
High-speed data network switches
IBM Storage Scale System components also require a high-speed data network IP or InfiniBand connection for the reading and writing of user data. As part of IBM Storage Scale System testing, IBM tests with Nvidia data network switches.
As a best practice, order and use the qualified data network switches where possible. These switches are tested and integrated with the IBM Storage Scale System solution.
Nvidia high-speed network switch drivers and firmware are tested and delivered integrated within the IBM Storage Scale System software solution stack.
Alternatively, the client can provide their own high-speed data network switch infrastructure. In this case, the client is responsible for provisioning networking port counts, firmware, network configuration, and definitions. The client is also responsible for maintaining and troubleshooting any networking issues.
Racking
The IBM Storage Scale System solution can be ordered with or without the building block being integrated into an IBM rack.
An IBM Storage Scale System solution that is ordered with feature codes that specify integration into the rack in IBM manufacturing results in a fully integrated, tested, and pre-cabled solution.
The IBM 5149-R42 is the default for most models of IBM Storage Scale Systems. The rack provides 42 EIA (42U) of usable space for IBM Storage Scale System components and network switches. A choice of various electrical PDUs is available to be specified for inclusion in the rack side areas.
It is advantageous to order IBM Storage Scale System as a racked system because IBM Storage Scale System Manufacturing delivers an IBM Storage Scale System with the rack mostly pre-cabled and pre-tested. This configuration provides faster time to install with higher quality.
If installing the IBM Storage Scale System in a customer-supplied rack, it is the customer’s responsibility to correctly specify all physical planning for electrical power, connectors, cables, and racking locations.
Clients also must plan the physical installation and cabling of the IBM Storage Scale System components in the client-supplied rack. An IBM TSS Services contract is available and recommended to perform physical planning, specification, and physical installation of the IBM Storage Scale System components in a client rack on behalf of the client.
For the rackless IBM Storage Scale System order, the solution must be assembled and integrated into the rack at the client site.
For the purposes of cable management and serviceability, each IBM Storage Scale System building block is integrated and deployed within the same rack.
If you need to split the IBM Storage Scale System components between adjacent racks, ask your IBM or IBM Business Partner representative to submit an RPQ special bid request. IBM examines the request and responds regarding whether IBM can grant and support the specific requested physical racking.
Software components
This section describes the software components that are used on the IBM Storage Scale System solution.
Software solution stack overview
The IBM Storage Scale System solution provides an integrated and tested stack of these operating system, adapter drivers, firmware, IBM Storage Scale software, management software, and installation scripts into a full IBM Storage Scale System software stack. This software solution stack is supported as an integrated IBM Storage Scale storage building block solution by IBM Service and Support.
The following sections includes descriptions of the following IBM Storage Scale System solution stack software components:
- RHEL operating system.
- IBM Storage Scale high-performance parallel file system.
- IBM Storage Scale RAID software-defined storage (SDS).
- IBM Storage Scale System solution installation and management software, which includes (but is not limited to) the following components:
- IBM Storage Scale System-specific documentation for installation and upgrade scripts
- Other tools for the IBM System Services Representative (IBM SSR) to use while installing IBM Storage Scale System, such as
essinstallcheck ,essnetverify , andessutils
- Third-generation IBM Storage Scale System systems deploy a container-oriented management software stack in the IBM Storage Scale System Management Server that includes Red Hat Ansible playbooks for enhanced ease of use during installation and orchestration. Installation is faster, and skill requirements are significantly lower than previous-generation IBM Storage Scale System models.
The IBM Storage Scale System solution stack levels are released as a version, release, and modification level.
The IBM Storage Scale System solution components are periodically upgraded to newer release levels, tested as an integrated solution, and released as a new level of the IBM Storage Scale System solution software. Additionally, generalized recommendations are made available to help clients implement a code update strategy. It is a full-field perspective, and as such, a customized recommendation that accounts for specifics such as business upgrade windows, length of time since last update, and decommission plans. You might require assistance from local support teams. For more information, see IBM Storage Scale Software Version Recommendation Preventive Service Planning.
Operating system
The IBM Storage Scale System solution runs RHEL as the operating system on the IBM Storage Scale Data Servers.
Each IBM Storage Scale System solution release level integrates and tests a suitable current level of RHEL, including any necessary RHEL fixes and errata that are required for the successful operation of the IBM Storage Scale System solution stack. IBM periodically provides new IBM Storage Scale System solution release levels that incorporate newer levels of RHEL. These releases are provided often enough to assure that a current level of RHEL is always available.
IBM uses an embedded RHEL license. This license is included with every order.
IBM Storage Scale
This section provides a description of IBM Storage Scale, the high-performance parallel clustered file system software that is used in IBM Storage Scale System. IBM Storage Scale software is released several months before the equivalent IBM Storage Scale System release. Intentionally, the release numbers of the Storage Scale software differ in the first digit from the Storage Scale System software. At the time of publication, the current software version is 6.0.0. The current Storage Scale System version is 7.0.0.
IBM Storage Scale is highly distributed, clustered file system software that provides high-speed concurrent data access to applications that run on multiple nodes and clusters. In addition to providing parallel high-performance file storage capabilities at exabyte scale, IBM Storage Scale provides tools for tiering, management, administration, and archiving of that enterprise-level data. IBM Storage Scale is the IBM strategic SDS for enterprise big data, analytics, and AI applications.
In the most common IBM Storage Scale deployment architecture, IBM Storage Scale data is accessed by IBM Storage Scale clients and users over a LAN network, accessing disk volumes that are known as Network Shared Disk (NSDs). The NSDs are attached to IBM Storage Scale nodes that are known as NSD Data Servers. In this IBM Redpaper publication, these nodes are also referred to as Data Servers.
IBM Storage Scale RAID
IBM Storage Scale RAID is an SDS controller that performs all of the storage controller functions that are normally associated with hardware storage controllers. IBM Storage Scale RAID integrates all HA and features of an advanced storage server into IBM Storage Scale SDS.
IBM Storage Scale RAID runs on the IBM Storage Scale System NSD Data Servers. IBM Storage Scale RAID provides sophisticated data placement and error correction algorithms to deliver high levels of storage reliability, availability, and serviceability, and performance.
IBM Storage Scale RAID implements a declustered erasure code parity schema, distributing data, redundancy information, and spare space across all disks of the IBM Storage Scale System enclosures. With this approach, a significant improvement is realized on the application performance. Also, storage rebuild time overhead is reduced (disk failure recovery process) compared to conventional RAID controllers.
IBM Storage Scale RAID implements large cache for performance by using memory on the IBM Storage Scale System NSD Data Servers. The large cache intelligently improves read and write performance, particularly for small block I/O operations.
IBM Storage Scale RAID mitigates performance effects of storage rebuilds
If storage failures occur, IBM Storage Scale RAID reconstructs lost or erased stripes for I/O operations dynamically. By using the highly distributed erasure coding, IBM Storage Scale RAID mitigates the performance effect of storage media failures.
IBM Storage Scale RAID end-to-end checksums
IBM Storage Scale RAID includes integrated end-to-end checksums that detect data corruption that might otherwise go undetected by a conventional storage controller. Unlike conventional storage controllers, IBM Storage Scale RAID is integrated with the IBM Storage Scale file system and performs end-to-end checksum comparison all the way out to the IBM Storage Scale client code on the workstations. This feature ensures data integrity at a file system level, detecting and automatically correcting data corruption errors that might occur in conventional storage environments.
In an environment where a customer experienced excessive file system checks and suffered downtime to repair file systems, the use of IBM Storage Scale RAID end-to-end checksums mitigates file system check problems. This feature assures availability of data and removes application outages that are caused by file system checks.
IBM Storage Scale RAID disk hospital
One of the key features of IBM Storage Scale RAID is the disk hospital. This powerful function asynchronously diagnoses errors and faults in the IBM Storage Scale System storage media, down to the level of the individual drive and the individual performance of each drive.
IBM Storage Scale RAID is fully aware of and tracks the performance of each individual drive because not all drives perform equally. IBM Storage Scale RAID uses the individual performance history of each drive to make intelligent data allocation and data retrieval decisions.
Extensive health metrics down to the level of the individual drive are maintained by the disk hospital. Performance variation is continually monitored. If or when a disk metric exceeds a threshold, the storage media is marked for replacement according to the disk maintenance replacement policy for the declustered array.
As an example, disk hospital features the following metrics:
relativePerformance , which characterizes response times. Values are compared to the average speed. If the metric falls below a specific threshold, the hospital addsslow to the pdisk state, and the disk is prepared for replacement.dataBadness , which characterizes media errors (hard errors) and checksum errors.
For more information about IBM Storage Scale RAID implementation and best practices, see the Administering IBM Storage Scale RAID documentation.
IBM Storage Scale System solution installation and management scripts
This section includes an overview of the components of the IBM Storage Scale System solution installation and management scripts. This overview includes the following information:
- IBM Storage Scale System-specific documentation for installation and upgrade scripts
- Ansible playbooks
- IBM Storage Scale System specific tools and utilities for the IBM SSR and administrators to use while installing or maintaining the IBM Storage Scale System
For more information about these IBM Storage Scale System solution components, see the IBM Storage Scale System solution release-specific level information at IBM Storage Scale System Software documentation.
Installation, upgrade, and administering guides
IBM provides manuals and documentation for deploying and administering IBM Storage Scale System, including the following publications:
- Quick Deployment Guide (QDG), which documents IBM Storage Scale System-specific scripts for installing, deploying, and upgrading IBM Storage Scale System for experienced users
- IBM Storage Scale RAID Administration, which focuses on administering IBM Storage Scale RAID on IBM Storage Scale System
- Problem determination guide, which provides more information about monitoring, troubleshooting, and maintenance procedures
For more information, see IBM Storage Scale System Software documentation.
Ansible playbooks and container-based software installation
Third-generation IBM Storage Scale System models include a software update process and methodology that are based on Ansible playbooks.
On this IBM Storage Scale System Management Server node, Ansible is configured to store the container and cluster configuration and definitions for all the managed IBM Storage Scale System Data Servers. Network services (such as Dynamic Host Configuration Protocol (DHCP) and SSH) are enabled to allow IBM Storage Scale System Management Server to deploy IBM Storage Scale System solution stack components to all IBM Storage Scale System nodes in the cluster.
For more information about
Footnotes
-
IBM newsroom: IBM Storage Scale System 6000 delivers industry-leading throughput for data-intensive workloads. IBM, 2024. ↩