Skip to main content

SQOOP

Apache Sqoop
https://sqoop.apache.org/docs/1.99.7/index.html

What is Apache Sqoop?​

SQOOP = SQL + HADOOP​

(SQL-to-Hadoop)

It is a tool designed to support bulk export and import of data into HDFS from structured data stores such as relational databases, enterprise data warehouses, and NoSQL systems. It is a data migration tool based upon a connector architecture which supports plugins to provide connectivity to new external systems.

Key Features of SQOOP​

  • Parallel Import/Export: Sqoop Big Data Tool uses the YARN framework to import and export data. This provides fault tolerance on top of parallelism
  • Import Results of SQL Query: Big Data Hadoop Sqoop allows us to import the result returned from an SQL query into HDFS
  • Connectors for All Major RDBMS Databases: It provides connectors for multiple Relational Database Management System (RDBMS) databases such as MySQL and MS SQL Server
  • Offers Full and Incremental Load: It can load the whole table or parts of the table by a single command. Hence, it supports the full and incremental load.

Install SQOOP​

Sqoop Tools
Controlling the Hadoop Installation

su - hadoop

/home/hadoop:
Download and untar sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz .

.bashrc:

...
...
export HADOOP_HOME=/home/hadoop/hadoop-3.4.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

export SQOOP_HOME=/home/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0
export PATH=$PATH:$SQOOP_HOME/bin
source ./.bashrc

Verify Installation​

sqoop version

warning

hadoop@LL01:~$ sqoop version Warning: /home/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /home/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /home/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 2025-03-16 09:22:05,843 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 Sqoop 1.4.7

info

So, we'll install:

  • HBase
  • HCatalog
  • Accumulo
  • Zookeeper

sqoop help

...
...
Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information