Wednesday, June 1, 2016

sqoop examples

[cloudera@quickstart ~]$ sqoop import-all-tables \
    -m 1 \
    --connect jdbc:mysql://quickstart:3306/retail_db \
    --username=retail_dba \
    --password=cloudera \
    --compression-codec=snappy \
    --as-parquetfile \
    --warehouse-dir=/user/hive/warehouse \
    --hive-import


=========================================================

[cloudera@quickstart ~]$ ps -ef|grep mysql
root      1950     1  0 06:28 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql     2070  1950  0 06:28 ?        00:00:01 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
cloudera  7650  7486  0 07:03 pts/0    00:00:00 grep mysql
[cloudera@quickstart ~]$
[cloudera@quickstart ~]$
[cloudera@quickstart ~]$ sqoop import-all-tables \
>     -m 1 \
>     --connect jdbc:mysql://quickstart:3306/retail_db \
>     --username=retail_dba \
>     --password=cloudera \
>     --compression-codec=snappy \
>     --as-parquetfile \
>     --warehouse-dir=/user/hive/warehouse \
>     --hive-import
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/06/01 07:03:30 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
16/06/01 07:03:30 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/06/01 07:03:30 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
16/06/01 07:03:30 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
16/06/01 07:03:30 WARN tool.BaseSqoopTool: It seems that you're doing hive import directly into default
16/06/01 07:03:30 WARN tool.BaseSqoopTool: hive warehouse directory which is not supported. Sqoop is
16/06/01 07:03:30 WARN tool.BaseSqoopTool: firstly importing data into separate directory and then
16/06/01 07:03:30 WARN tool.BaseSqoopTool: inserting data into hive. Please consider removing
16/06/01 07:03:30 WARN tool.BaseSqoopTool: --target-dir or --warehouse-dir into /user/hive/warehouse in
16/06/01 07:03:30 WARN tool.BaseSqoopTool: case that you will detect any issues.
16/06/01 07:03:31 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
16/06/01 07:03:31 INFO tool.CodeGenTool: Beginning code generation
16/06/01 07:03:31 INFO tool.CodeGenTool: Will generate java class as codegen_categories
16/06/01 07:03:31 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
16/06/01 07:03:31 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
16/06/01 07:03:31 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/06/01 07:03:33 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.jar
16/06/01 07:03:33 WARN manager.MySQLManager: It looks like you are importing from mysql.
16/06/01 07:03:33 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
16/06/01 07:03:33 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
16/06/01 07:03:33 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
16/06/01 07:03:33 INFO mapreduce.ImportJobBase: Beginning import of categories
16/06/01 07:03:33 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
16/06/01 07:03:33 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
16/06/01 07:03:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
16/06/01 07:03:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
16/06/01 07:03:35 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:03:35 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:03:35 INFO hive.metastore: Connected to metastore.
16/06/01 07:03:35 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:03:35 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:03:35 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:03:35 INFO hive.metastore: Connected to metastore.
16/06/01 07:03:36 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:03:36 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:03:36 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:03:36 INFO hive.metastore: Connected to metastore.
16/06/01 07:03:36 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:03:36 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:03:36 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:03:36 INFO hive.metastore: Connected to metastore.
16/06/01 07:03:36 INFO hive.HiveManagedMetadataProvider: Creating a managed Hive table named: categories
16/06/01 07:03:37 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:03:37 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:03:37 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:03:37 INFO hive.metastore: Connected to metastore.
16/06/01 07:03:49 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
16/06/01 07:03:49 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/06/01 07:03:59 INFO db.DBInputFormat: Using read commited transaction isolation
16/06/01 07:03:59 INFO mapreduce.JobSubmitter: number of splits:1
16/06/01 07:03:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1464787780964_0001
16/06/01 07:04:00 INFO impl.YarnClientImpl: Submitted application application_1464787780964_0001
16/06/01 07:04:00 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1464787780964_0001/
16/06/01 07:04:00 INFO mapreduce.Job: Running job: job_1464787780964_0001
16/06/01 07:04:27 INFO mapreduce.Job: Job job_1464787780964_0001 running in uber mode : false
16/06/01 07:04:27 INFO mapreduce.Job:  map 0% reduce 0%
16/06/01 07:04:43 INFO mapreduce.Job:  map 100% reduce 0%
16/06/01 07:04:44 INFO mapreduce.Job: Job job_1464787780964_0001 completed successfully
16/06/01 07:04:44 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=211514
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=5685
        HDFS: Number of bytes written=3446
        HDFS: Number of read operations=48
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=10
    Job Counters
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=12540
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=12540
        Total vcore-seconds taken by all map tasks=12540
        Total megabyte-seconds taken by all map tasks=12840960
    Map-Reduce Framework
        Map input records=58
        Map output records=58
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=141
        CPU time spent (ms)=2640
        Physical memory (bytes) snapshot=336764928
        Virtual memory (bytes) snapshot=1592012800
        Total committed heap usage (bytes)=326107136
    File Input Format Counters
        Bytes Read=0
    File Output Format Counters
        Bytes Written=0
16/06/01 07:04:44 INFO mapreduce.ImportJobBase: Transferred 3.3652 KB in 55.2648 seconds (62.3543 bytes/sec)
16/06/01 07:04:44 INFO mapreduce.ImportJobBase: Retrieved 58 records.
16/06/01 07:04:44 INFO tool.CodeGenTool: Beginning code generation
16/06/01 07:04:44 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `customers` AS t LIMIT 1
16/06/01 07:04:44 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/06/01 07:04:49 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.jar
16/06/01 07:04:49 INFO mapreduce.ImportJobBase: Beginning import of customers
16/06/01 07:04:49 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
16/06/01 07:04:49 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `customers` AS t LIMIT 1
16/06/01 07:04:49 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `customers` AS t LIMIT 1
16/06/01 07:04:49 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:04:49 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:04:49 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:04:49 INFO hive.metastore: Connected to metastore.
16/06/01 07:04:50 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:04:50 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:04:50 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:04:50 INFO hive.metastore: Connected to metastore.
16/06/01 07:04:50 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:04:50 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:04:50 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:04:50 INFO hive.metastore: Connected to metastore.
16/06/01 07:04:50 INFO hive.HiveManagedMetadataProvider: Creating a managed Hive table named: customers
16/06/01 07:04:50 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:04:50 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:04:50 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:04:50 INFO hive.metastore: Connected to metastore.
16/06/01 07:04:50 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/06/01 07:05:00 INFO db.DBInputFormat: Using read commited transaction isolation
16/06/01 07:05:00 INFO mapreduce.JobSubmitter: number of splits:1
16/06/01 07:05:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1464787780964_0002
16/06/01 07:05:00 INFO impl.YarnClientImpl: Submitted application application_1464787780964_0002
16/06/01 07:05:00 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1464787780964_0002/
16/06/01 07:05:00 INFO mapreduce.Job: Running job: job_1464787780964_0002
16/06/01 07:05:27 INFO mapreduce.Job: Job job_1464787780964_0002 running in uber mode : false
16/06/01 07:05:27 INFO mapreduce.Job:  map 0% reduce 0%
16/06/01 07:05:44 INFO mapreduce.Job:  map 100% reduce 0%
16/06/01 07:05:49 INFO mapreduce.Job: Job job_1464787780964_0002 completed successfully
16/06/01 07:05:49 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=212281
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=13005
        HDFS: Number of bytes written=257967
        HDFS: Number of read operations=48
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=10
    Job Counters
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=13696
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=13696
        Total vcore-seconds taken by all map tasks=13696
        Total megabyte-seconds taken by all map tasks=14024704
    Map-Reduce Framework
        Map input records=12435
        Map output records=12435
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=148
        CPU time spent (ms)=5080
        Physical memory (bytes) snapshot=341532672
        Virtual memory (bytes) snapshot=1585790976
        Total committed heap usage (bytes)=268435456
    File Input Format Counters
        Bytes Read=0
    File Output Format Counters
        Bytes Written=0
16/06/01 07:05:49 INFO mapreduce.ImportJobBase: Transferred 251.9209 KB in 59.1842 seconds (4.2566 KB/sec)
16/06/01 07:05:49 INFO mapreduce.ImportJobBase: Retrieved 12435 records.
16/06/01 07:05:49 INFO tool.CodeGenTool: Beginning code generation
16/06/01 07:05:49 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `departments` AS t LIMIT 1
16/06/01 07:05:50 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/06/01 07:05:55 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.jar
16/06/01 07:05:55 INFO mapreduce.ImportJobBase: Beginning import of departments
16/06/01 07:05:55 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
16/06/01 07:05:55 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `departments` AS t LIMIT 1
16/06/01 07:05:55 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `departments` AS t LIMIT 1
16/06/01 07:05:55 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:05:55 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:05:55 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:05:55 INFO hive.metastore: Connected to metastore.
16/06/01 07:05:55 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:05:55 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:05:56 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:05:56 INFO hive.metastore: Connected to metastore.
16/06/01 07:05:56 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:05:56 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:05:56 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:05:56 INFO hive.metastore: Connected to metastore.
16/06/01 07:05:56 INFO hive.HiveManagedMetadataProvider: Creating a managed Hive table named: departments
16/06/01 07:05:56 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:05:56 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:05:56 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:05:56 INFO hive.metastore: Connected to metastore.
16/06/01 07:05:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/06/01 07:06:06 INFO db.DBInputFormat: Using read commited transaction isolation
16/06/01 07:06:06 INFO mapreduce.JobSubmitter: number of splits:1
16/06/01 07:06:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1464787780964_0003
16/06/01 07:06:06 INFO impl.YarnClientImpl: Submitted application application_1464787780964_0003
16/06/01 07:06:06 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1464787780964_0003/
16/06/01 07:06:06 INFO mapreduce.Job: Running job: job_1464787780964_0003
16/06/01 07:06:35 INFO mapreduce.Job: Job job_1464787780964_0003 running in uber mode : false
16/06/01 07:06:35 INFO mapreduce.Job:  map 0% reduce 0%
16/06/01 07:08:20 INFO mapreduce.Job:  map 100% reduce 0%
16/06/01 07:08:33 INFO mapreduce.Job: Job job_1464787780964_0003 completed successfully
16/06/01 07:08:34 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=211387
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=4453
        HDFS: Number of bytes written=1982
        HDFS: Number of read operations=48
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=10
    Job Counters
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=105084
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=105084
        Total vcore-seconds taken by all map tasks=105084
        Total megabyte-seconds taken by all map tasks=107606016
    Map-Reduce Framework
        Map input records=6
        Map output records=6
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=1788
        CPU time spent (ms)=3220
        Physical memory (bytes) snapshot=253460480
        Virtual memory (bytes) snapshot=1580556288
        Total committed heap usage (bytes)=267911168
    File Input Format Counters
        Bytes Read=0
    File Output Format Counters
        Bytes Written=0
16/06/01 07:08:34 INFO mapreduce.ImportJobBase: Transferred 1.9355 KB in 157.3586 seconds (12.5954 bytes/sec)
16/06/01 07:08:34 INFO mapreduce.ImportJobBase: Retrieved 6 records.
16/06/01 07:08:34 INFO tool.CodeGenTool: Beginning code generation
16/06/01 07:08:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `order_items` AS t LIMIT 1
16/06/01 07:08:34 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/06/01 07:08:39 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.jar
16/06/01 07:08:39 INFO mapreduce.ImportJobBase: Beginning import of order_items
16/06/01 07:08:39 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
16/06/01 07:08:39 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `order_items` AS t LIMIT 1
16/06/01 07:08:39 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `order_items` AS t LIMIT 1
16/06/01 07:08:39 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:08:39 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:08:39 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:08:39 INFO hive.metastore: Connected to metastore.
16/06/01 07:08:40 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:08:40 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:08:40 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:08:40 INFO hive.metastore: Connected to metastore.
16/06/01 07:08:40 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:08:40 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:08:40 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:08:40 INFO hive.metastore: Connected to metastore.
16/06/01 07:08:40 INFO hive.HiveManagedMetadataProvider: Creating a managed Hive table named: order_items
16/06/01 07:08:40 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:08:40 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:08:40 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:08:40 INFO hive.metastore: Connected to metastore.
16/06/01 07:08:40 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/06/01 07:08:49 INFO db.DBInputFormat: Using read commited transaction isolation
16/06/01 07:08:49 INFO mapreduce.JobSubmitter: number of splits:1
16/06/01 07:08:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1464787780964_0004
16/06/01 07:08:49 INFO impl.YarnClientImpl: Submitted application application_1464787780964_0004
16/06/01 07:08:49 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1464787780964_0004/
16/06/01 07:08:49 INFO mapreduce.Job: Running job: job_1464787780964_0004
16/06/01 07:09:23 INFO mapreduce.Job: Job job_1464787780964_0004 running in uber mode : false
16/06/01 07:09:23 INFO mapreduce.Job:  map 0% reduce 0%
16/06/01 07:09:56 INFO mapreduce.Job:  map 100% reduce 0%
16/06/01 07:10:20 INFO mapreduce.Job: Job job_1464787780964_0004 completed successfully
16/06/01 07:10:21 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=211981
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=9725
        HDFS: Number of bytes written=1649632
        HDFS: Number of read operations=48
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=10
    Job Counters
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=46491
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=46491
        Total vcore-seconds taken by all map tasks=46491
        Total megabyte-seconds taken by all map tasks=47606784
    Map-Reduce Framework
        Map input records=172198
        Map output records=172198
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=741
        CPU time spent (ms)=6910
        Physical memory (bytes) snapshot=412913664
        Virtual memory (bytes) snapshot=1575092224
        Total committed heap usage (bytes)=342360064
    File Input Format Counters
        Bytes Read=0
    File Output Format Counters
        Bytes Written=0
16/06/01 07:10:21 INFO mapreduce.ImportJobBase: Transferred 1.5732 MB in 100.2474 seconds (16.0699 KB/sec)
16/06/01 07:10:21 INFO mapreduce.ImportJobBase: Retrieved 172198 records.
16/06/01 07:10:21 INFO tool.CodeGenTool: Beginning code generation
16/06/01 07:10:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `orders` AS t LIMIT 1
16/06/01 07:10:21 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/06/01 07:10:27 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.jar
16/06/01 07:10:27 INFO mapreduce.ImportJobBase: Beginning import of orders
16/06/01 07:10:27 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
16/06/01 07:10:27 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `orders` AS t LIMIT 1
16/06/01 07:10:27 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `orders` AS t LIMIT 1
16/06/01 07:10:27 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:10:27 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:10:27 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:10:27 INFO hive.metastore: Connected to metastore.
16/06/01 07:10:27 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:10:27 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:10:27 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:10:27 INFO hive.metastore: Connected to metastore.
16/06/01 07:10:28 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:10:28 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:10:28 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:10:28 INFO hive.metastore: Connected to metastore.
16/06/01 07:10:28 INFO hive.HiveManagedMetadataProvider: Creating a managed Hive table named: orders
16/06/01 07:10:28 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:10:28 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:10:28 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:10:28 INFO hive.metastore: Connected to metastore.
16/06/01 07:10:28 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/06/01 07:10:37 INFO db.DBInputFormat: Using read commited transaction isolation
16/06/01 07:10:37 INFO mapreduce.JobSubmitter: number of splits:1
16/06/01 07:10:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1464787780964_0005
16/06/01 07:10:38 INFO impl.YarnClientImpl: Submitted application application_1464787780964_0005
16/06/01 07:10:38 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1464787780964_0005/
16/06/01 07:10:38 INFO mapreduce.Job: Running job: job_1464787780964_0005
16/06/01 07:11:10 INFO mapreduce.Job: Job job_1464787780964_0005 running in uber mode : false
16/06/01 07:11:10 INFO mapreduce.Job:  map 0% reduce 0%
16/06/01 07:11:33 INFO mapreduce.Job:  map 100% reduce 0%
16/06/01 07:11:45 INFO mapreduce.Job: Job job_1464787780964_0005 completed successfully
16/06/01 07:11:46 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=211574
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=6589
        HDFS: Number of bytes written=489972
        HDFS: Number of read operations=48
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=10
    Job Counters
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=23779
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=23779
        Total vcore-seconds taken by all map tasks=23779
        Total megabyte-seconds taken by all map tasks=24349696
    Map-Reduce Framework
        Map input records=68883
        Map output records=68883
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=780
        CPU time spent (ms)=6320
        Physical memory (bytes) snapshot=398024704
        Virtual memory (bytes) snapshot=1588559872
        Total committed heap usage (bytes)=342360064
    File Input Format Counters
        Bytes Read=0
    File Output Format Counters
        Bytes Written=0
16/06/01 07:11:46 INFO mapreduce.ImportJobBase: Transferred 478.4883 KB in 77.3825 seconds (6.1834 KB/sec)
16/06/01 07:11:46 INFO mapreduce.ImportJobBase: Retrieved 68883 records.
16/06/01 07:11:46 INFO tool.CodeGenTool: Beginning code generation
16/06/01 07:11:46 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `products` AS t LIMIT 1
16/06/01 07:11:46 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/06/01 07:11:53 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/9c5d81b8a4fc931d02a97efaa7fd3d8a/codegen_categories.jar
16/06/01 07:11:53 INFO mapreduce.ImportJobBase: Beginning import of products
16/06/01 07:11:53 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
16/06/01 07:11:53 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `products` AS t LIMIT 1
16/06/01 07:11:53 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `products` AS t LIMIT 1
16/06/01 07:11:54 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:11:54 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:11:54 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:11:54 INFO hive.metastore: Connected to metastore.
16/06/01 07:11:54 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:11:54 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:11:54 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:11:54 INFO hive.metastore: Connected to metastore.
16/06/01 07:11:54 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:11:54 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:11:54 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:11:54 INFO hive.metastore: Connected to metastore.
16/06/01 07:11:54 INFO hive.HiveManagedMetadataProvider: Creating a managed Hive table named: products
16/06/01 07:11:54 INFO hive.metastore: Closed a connection to metastore, current connections: 0
16/06/01 07:11:54 INFO hive.metastore: Trying to connect to metastore with URI thrift://127.0.0.1:9083
16/06/01 07:11:54 INFO hive.metastore: Opened a connection to metastore, current connections: 1
16/06/01 07:11:54 INFO hive.metastore: Connected to metastore.
16/06/01 07:11:55 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/06/01 07:12:04 INFO db.DBInputFormat: Using read commited transaction isolation
16/06/01 07:12:04 INFO mapreduce.JobSubmitter: number of splits:1
16/06/01 07:12:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1464787780964_0006
16/06/01 07:12:04 INFO impl.YarnClientImpl: Submitted application application_1464787780964_0006
16/06/01 07:12:04 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1464787780964_0006/
16/06/01 07:12:04 INFO mapreduce.Job: Running job: job_1464787780964_0006
16/06/01 07:12:39 INFO mapreduce.Job: Job job_1464787780964_0006 running in uber mode : false
16/06/01 07:12:39 INFO mapreduce.Job:  map 0% reduce 0%
16/06/01 07:13:14 INFO mapreduce.Job:  map 100% reduce 0%
16/06/01 07:13:15 INFO mapreduce.Job: Job job_1464787780964_0006 completed successfully
16/06/01 07:13:15 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=211879
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=9261
        HDFS: Number of bytes written=47239
        HDFS: Number of read operations=48
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=10
    Job Counters
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=22970
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=22970
        Total vcore-seconds taken by all map tasks=22970
        Total megabyte-seconds taken by all map tasks=23521280
    Map-Reduce Framework
        Map input records=1345
        Map output records=1345
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=119
        CPU time spent (ms)=3040
        Physical memory (bytes) snapshot=298315776
        Virtual memory (bytes) snapshot=1590517760
        Total committed heap usage (bytes)=185597952
    File Input Format Counters
        Bytes Read=0
    File Output Format Counters
        Bytes Written=0
16/06/01 07:13:15 INFO mapreduce.ImportJobBase: Transferred 46.1318 KB in 80.5216 seconds (586.6626 bytes/sec)
16/06/01 07:13:15 INFO mapreduce.ImportJobBase: Retrieved 1345 records.

No comments:

Post a Comment