Skip to main content

Enterprise Data Warehouse Optimization with Hadoop on IBM Power Systems Servers

An IBM Redpaper publication

thumbnail 

Published on 31 January 2018

  1. .EPUB (1.2 MB)
  2. .PDF (2.5 MB)

Apple BooksGoogle Play Books

Share this page:   

ISBN-10: 0738456608
ISBN-13: 9780738456607
IBM Form #: REDP-5476-00


Authors: Scott Vetter, Helen Lu and Maciej Olejniczak

    menu icon

    Abstract

    Data warehouses were developed for many good reasons, such as providing quick query and reporting for business operations, and business performance. However, over the years, due to the explosion of applications and data volume, many existing data warehouses have become difficult to manage. Extract, Transform, and Load (ETL) processes are taking longer, missing their allocated batch windows. In addition, data types that are required for business analysis have expanded from structured data to unstructured data.

    The Apache open source Hadoop platform provides a great alternative for solving these problems.

    IBM® has committed to open source since the early years of open Linux. IBM and Hortonworks together are committed to Apache open source software more than any other company.

    IBM Power Systems™ servers are built with open technologies and are designed for mission-critical data applications. Power Systems servers use technology from the OpenPOWER Foundation, an open technology infrastructure that uses the IBM POWER® architecture to help meet the evolving needs of big data applications. The combination of Power Systems with Hortonworks Data Platform (HDP) provides users with a highly efficient platform that provides leadership performance for big data workloads such as Hadoop and Spark.

    This IBM Redpaper™ publication provides details about Enterprise Data Warehouse (EDW) optimization with Hadoop on Power Systems. Many people know Power Systems from the IBM AIX® platform, but might not be familiar with IBM PowerLinux™, so part of this paper provides a Power Systems overview. A quick introduction to Hadoop is provided for those not familiar with the topic. Details of HDP on Power Reference architecture are included that will help both software architects and infrastructure architects understand the design.

    In the optimization chapter, we describe various topics: traditional EDW offload, sizing guidelines, performance tuning, IBM Elastic Storage™ Server (ESS) for data-intensive workload, IBM Big SQL as the common structured query language (SQL) engine for Hadoop platform, and tools that are available on Power Systems that are related to EDW optimization. We also dedicate some pages to the analytics components (IBM Data Science Experience (IBM DSX) and IBM Spectrum™ Conductor for Spark workload) for the Hadoop infrastructure.

    Table of Contents

    Chapter 1. Enterprise Data Warehouse overview

    Chapter 2. IBM Power Systems overview

    Chapter 3. Hortonworks Data Platform on IBM Power Systems reference architecture

    Chapter 4. Enterprise Data Warehouse on Hadoop optimization

     

    Others who read this also read