Hadoop for Developers

Overview

Apache Hadoop is the most popular framework for processing Big Data on clusters of servers. This course will introduce a developer to the Hadoop ecosystem.

Objective

Learn Hadoop Ecosystem and tools for Big Data analytics

What You Will Learn

  • Hadoop & Big Data
  • HDFS
  • YARN
  • Hive
  • HBase

Audience:

Developers

Duration:

Three days

Format :

Lectures and hands-on labs. (50% – 50%)

Prerequisites

  • Developer background
  • Comfortable with SQL and Java programming language (HBase labs are in Java)
  • Comfortable in a Linux environment

Lab environment

Zero Install: There is no need to install Hadoop software on students’ machines! A working Hadoop cluster will be provided for students.

Students will need the following

  • An SSH client (Linux and Mac already have ssh clients, for Windows Putty is recommended)
  • A browser to access the cluster. We recommend Chrome browser

Detailed outline

Introduction to Hadoop

    • Hadoop Ecosystem
    • Hadoop Distributions
    • High-level architecture
    • Hardware and software
    • Lab: first look at Hadoop

HDFS

    • Design and architecture
    • Concepts (horizontal scaling, replication, data locality, rack awareness)
    • Daemons: Namenode, Secondary Namenode, Datanode
    • Communications and heart-beats
    • Data integrity
    • Read and write path
    • Namenode High Availability (HA), Federation
    • Labs: Interacting with HDFS

YARN

    • YARN Concepts and architecture
    • Resource Manager, Node Manager
    • Writing YARN applications
    • Labs: Running a sample YARN program

Hive

    • Architecture and design
    • Hive Data types
    • HQL
    • Creating Hive tables and querying
    • Partitions
    • Joins
    • Text processing
    • Labs: various labs on processing data with Hive

HBase

    • Concepts and architecture
    • HBase vs RDBMS vs Cassandra
    • HBase Java API
    • Time series data on HBase
    • Schema design
    • Labs: Interacting with HBase using shell;   programming in HBase Java API ; Schema design exercise