Monday, December 4, 2017

Impala vs Hive(MR) vs Hive(Spark)

Query 1

(First Execution)

Query 1

(verify Caching)

Query 2

(Same Base Table)

Impala

5.84s

0.15s

0.44s

Hive on MR2

26.288s

24.367s

53.177s

Hive on SPark

31.798s

3.284s

3.407s

[donghua@cdh-vm ~]$ impala-shell -i cdh-vm.dbaglobe.com -d default
Starting Impala Shell without Kerberos authentication
Connected to cdh-vm.dbaglobe.com:21000
Server version: impalad version 2.10.0-cdh5.13.0 RELEASE (build 2511805f1eaa991df1460276c7e9f19d819cd4e4)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.0 (2511805) built on Wed Oct  4 10:55:37 PDT 2017)

To see more tips, run the TIP command.
***********************************************************************************
Query: use `default`
[cdh-vm.dbaglobe.com:21000] > show tables;
Query: show tables
Fetched 0 row(s) in 0.00s
[cdh-vm.dbaglobe.com:21000] > invalidate metadata;
Query: invalidate metadata
Query submitted at: 2017-12-04 21:47:45 (Coordinator:
http://cdh-vm.dbaglobe.com:25000)
Query progress can be monitored at: http://cdh-vm.dbaglobe.com:25000/query_plan?query_id=b04a346400d38fed:21d3756a00000000
Fetched 0 row(s) in 3.64s
[cdh-vm.dbaglobe.com:21000] > show tables;
Query: show tables
+-------------+
| name        |
+-------------+
| iotdatademo |
+-------------+
Fetched 1 row(s) in 0.00s
[cdh-vm.dbaglobe.com:21000] > select count(*) from iotdatademo;
Query: select count(*) from iotdatademo
Query submitted at: 2017-12-04 21:48:06 (Coordinator:
http://cdh-vm.dbaglobe.com:25000)
Query progress can be monitored at: http://cdh-vm.dbaglobe.com:25000/query_plan?query_id=8c4202fcb2d7bfc4:891ac4a100000000
+----------+
| count(*) |
+----------+
| 864010   |
+----------+
Fetched 1 row(s) in 5.84s
[cdh-vm.dbaglobe.com:21000] > select count(*) from iotdatademo;
Query: select count(*) from iotdatademo
Query submitted at: 2017-12-04 21:48:14 (Coordinator:
http://cdh-vm.dbaglobe.com:25000)
Query progress can be monitored at: http://cdh-vm.dbaglobe.com:25000/query_plan?query_id=d433089da3461f8:7b786b700000000
+----------+
| count(*) |
+----------+
| 864010   |
+----------+
Fetched 1 row(s) in 0.15s
[cdh-vm.dbaglobe.com:21000] > select eday, avg(speed) from iotdatademo group by eday order by eday;
Query: select eday, avg(speed) from iotdatademo group by eday order by eday
Query submitted at: 2017-12-04 21:48:41 (Coordinator:
http://cdh-vm.dbaglobe.com:25000)
Query progress can be monitored at: http://cdh-vm.dbaglobe.com:25000/query_plan?query_id=7b4945e5560d6f73:cf15702700000000
+------+-------------------+
| eday | avg(speed)        |
+------+-------------------+
| 0    | 80.42217204861018 |
| 1    | 80.4242077305864  |
| 2    | 80.516892013888   |
| 3    | 80.42997673611163 |
| 4    | 80.62740798611237 |
| 5    | 80.49621712963015 |
| 6    | 80.54539832175986 |
+------+-------------------+
Fetched 7 row(s) in 0.44s
[cdh-vm.dbaglobe.com:21000] > exit;
Goodbye donghua
[donghua@cdh-vm ~]$


[donghua@cdh-vm ~]$ beeline -n donghua -u jdbc:hive2://cdh-vm:10000
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
scan complete in 3ms
Connecting to jdbc:hive2://cdh-vm:10000
Connected to: Apache Hive (version 1.1.0-cdh5.13.0)
Driver: Hive JDBC (version 1.1.0-cdh5.13.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.1.0-cdh5.13.0 by Apache Hive
0: jdbc:hive2://cdh-vm:10000> select count(*) from iotdatademo;
INFO  : Compiling command(queryId=hive_20171204220606_0f8b9bb5-5da0-415a-bf7b-d3e7068b031e): select count(*) from iotdatademo
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20171204220606_0f8b9bb5-5da0-415a-bf7b-d3e7068b031e); Time taken: 0.178 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20171204220606_0f8b9bb5-5da0-415a-bf7b-d3e7068b031e): select count(*) from iotdatademo
INFO  : Query ID = hive_20171204220606_0f8b9bb5-5da0-415a-bf7b-d3e7068b031e
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Number of reduce tasks determined at compile time: 1
INFO  : In order to change the average load for a reducer (in bytes):
INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
INFO  : In order to limit the maximum number of reducers:
INFO  :   set hive.exec.reducers.max=<number>
INFO  : In order to set a constant number of reducers:
INFO  :   set mapreduce.job.reduces=<number>
INFO  : number of splits:1
INFO  : Submitting tokens for job: job_1512396061167_0004
INFO  : The url to track the job:
http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0004/
INFO  : Starting Job = job_1512396061167_0004, Tracking URL = http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0004/
INFO  : Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1512396061167_0004
INFO  : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
INFO  : 2017-12-04 22:06:18,607 Stage-1 map = 0%,  reduce = 0%
INFO  : 2017-12-04 22:06:27,435 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.28 sec
INFO  : 2017-12-04 22:06:34,880 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 4.79 sec
INFO  : MapReduce Total cumulative CPU time: 4 seconds 790 msec
INFO  : Ended Job = job_1512396061167_0004
INFO  : MapReduce Jobs Launched:
INFO  : Stage-Stage-1: Map: 1  Reduce: 1   Cumulative CPU: 4.79 sec   HDFS Read: 46846177 HDFS Write: 7 SUCCESS
INFO  : Total MapReduce CPU Time Spent: 4 seconds 790 msec
INFO  : Completed executing command(queryId=hive_20171204220606_0f8b9bb5-5da0-415a-bf7b-d3e7068b031e); Time taken: 25.948 seconds
INFO  : OK
+---------+--+
|   _c0   |
+---------+--+
| 864010  |
+---------+--+
1 row selected (26.288 seconds)
0: jdbc:hive2://cdh-vm:10000> select count(*) from iotdatademo;
INFO  : Compiling command(queryId=hive_20171204220707_172c5f7b-7401-4a1e-aadf-f206f7037b32): select count(*) from iotdatademo
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20171204220707_172c5f7b-7401-4a1e-aadf-f206f7037b32); Time taken: 0.069 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20171204220707_172c5f7b-7401-4a1e-aadf-f206f7037b32): select count(*) from iotdatademo
INFO  : Query ID = hive_20171204220707_172c5f7b-7401-4a1e-aadf-f206f7037b32
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Number of reduce tasks determined at compile time: 1
INFO  : In order to change the average load for a reducer (in bytes):
INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
INFO  : In order to limit the maximum number of reducers:
INFO  :   set hive.exec.reducers.max=<number>
INFO  : In order to set a constant number of reducers:
INFO  :   set mapreduce.job.reduces=<number>
INFO  : number of splits:1
INFO  : Submitting tokens for job: job_1512396061167_0005
INFO  : The url to track the job:
http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0005/
INFO  : Starting Job = job_1512396061167_0005, Tracking URL = http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0005/
INFO  : Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1512396061167_0005
INFO  : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
INFO  : 2017-12-04 22:07:54,245 Stage-1 map = 0%,  reduce = 0%
INFO  : 2017-12-04 22:08:00,581 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.32 sec
INFO  : 2017-12-04 22:08:07,322 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 4.42 sec
INFO  : MapReduce Total cumulative CPU time: 4 seconds 420 msec
INFO  : Ended Job = job_1512396061167_0005
INFO  : MapReduce Jobs Launched:
INFO  : Stage-Stage-1: Map: 1  Reduce: 1   Cumulative CPU: 4.42 sec   HDFS Read: 46846177 HDFS Write: 7 SUCCESS
INFO  : Total MapReduce CPU Time Spent: 4 seconds 420 msec
INFO  : Completed executing command(queryId=hive_20171204220707_172c5f7b-7401-4a1e-aadf-f206f7037b32); Time taken: 24.253 seconds
INFO  : OK
+---------+--+
|   _c0   |
+---------+--+
| 864010  |
+---------+--+
1 row selected (24.367 seconds)
0: jdbc:hive2://cdh-vm:10000> select eday, avg(speed) from iotdatademo group by eday order by eday;
INFO  : Compiling command(queryId=hive_20171204220808_06c3d474-3fec-4c3a-b6ae-1771278b5080): select eday, avg(speed) from iotdatademo group by eday order by eday
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:eday, type:bigint, comment:null), FieldSchema(name:_c1, type:double, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20171204220808_06c3d474-3fec-4c3a-b6ae-1771278b5080); Time taken: 0.114 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20171204220808_06c3d474-3fec-4c3a-b6ae-1771278b5080): select eday, avg(speed) from iotdatademo group by eday order by eday
INFO  : Query ID = hive_20171204220808_06c3d474-3fec-4c3a-b6ae-1771278b5080
INFO  : Total jobs = 2
INFO  : Launching Job 1 out of 2
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Number of reduce tasks not specified. Estimated from input data size: 1
INFO  : In order to change the average load for a reducer (in bytes):
INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
INFO  : In order to limit the maximum number of reducers:
INFO  :   set hive.exec.reducers.max=<number>
INFO  : In order to set a constant number of reducers:
INFO  :   set mapreduce.job.reduces=<number>
INFO  : number of splits:1
INFO  : Submitting tokens for job: job_1512396061167_0006
INFO  : The url to track the job:
http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0006/
INFO  : Starting Job = job_1512396061167_0006, Tracking URL = http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0006/
INFO  : Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1512396061167_0006
INFO  : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
INFO  : 2017-12-04 22:08:27,407 Stage-1 map = 0%,  reduce = 0%
INFO  : 2017-12-04 22:08:34,171 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 3.52 sec
INFO  : 2017-12-04 22:08:41,696 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 5.93 sec
INFO  : MapReduce Total cumulative CPU time: 5 seconds 930 msec
INFO  : Ended Job = job_1512396061167_0006
INFO  : Launching Job 2 out of 2
INFO  : Starting task [Stage-2:MAPRED] in serial mode
INFO  : Number of reduce tasks determined at compile time: 1
INFO  : In order to change the average load for a reducer (in bytes):
INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
INFO  : In order to limit the maximum number of reducers:
INFO  :   set hive.exec.reducers.max=<number>
INFO  : In order to set a constant number of reducers:
INFO  :   set mapreduce.job.reduces=<number>
INFO  : number of splits:1
INFO  : Submitting tokens for job: job_1512396061167_0007
INFO  : The url to track the job:
http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0007/
INFO  : Starting Job = job_1512396061167_0007, Tracking URL = http://cdh-vm.dbaglobe.com:8088/proxy/application_1512396061167_0007/
INFO  : Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1512396061167_0007
INFO  : Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
INFO  : 2017-12-04 22:08:54,478 Stage-2 map = 0%,  reduce = 0%
INFO  : 2017-12-04 22:09:00,934 Stage-2 map = 100%,  reduce = 0%, Cumulative CPU 1.16 sec
INFO  : 2017-12-04 22:09:09,771 Stage-2 map = 100%,  reduce = 100%, Cumulative CPU 3.71 sec
INFO  : MapReduce Total cumulative CPU time: 3 seconds 710 msec
INFO  : Ended Job = job_1512396061167_0007
INFO  : MapReduce Jobs Launched:
INFO  : Stage-Stage-1: Map: 1  Reduce: 1   Cumulative CPU: 5.93 sec   HDFS Read: 46845921 HDFS Write: 278 SUCCESS
INFO  : Stage-Stage-2: Map: 1  Reduce: 1   Cumulative CPU: 3.71 sec   HDFS Read: 5188 HDFS Write: 137 SUCCESS
INFO  : Total MapReduce CPU Time Spent: 9 seconds 640 msec
INFO  : Completed executing command(queryId=hive_20171204220808_06c3d474-3fec-4c3a-b6ae-1771278b5080); Time taken: 53.003 seconds
INFO  : OK
+-------+--------------------+--+
| eday  |        _c1         |
+-------+--------------------+--+
| 0     | 80.42217204861018  |
| 1     | 80.4242077305864   |
| 2     | 80.516892013888    |
| 3     | 80.42997673611163  |
| 4     | 80.62740798611237  |
| 5     | 80.49621712963015  |
| 6     | 80.54539832175986  |
+-------+--------------------+--+
7 rows selected (53.177 seconds)
0: jdbc:hive2://cdh-vm:10000> !exit
Closing: 0: jdbc:hive2://cdh-vm:10000
[donghua@cdh-vm ~]$

[donghua@cdh-vm ~]$ beeline -n donghua -u jdbc:hive2://cdh-vm:10000
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
scan complete in 3ms
Connecting to jdbc:hive2://cdh-vm:10000
Connected to: Apache Hive (version 1.1.0-cdh5.13.0)
Driver: Hive JDBC (version 1.1.0-cdh5.13.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.1.0-cdh5.13.0 by Apache Hive
0: jdbc:hive2://cdh-vm:10000> SET hive.execution.engine=spark;
No rows affected (0.171 seconds)
0: jdbc:hive2://cdh-vm:10000> select count(*) from iotdatademo;
INFO  : Compiling command(queryId=hive_20171204222828_f697cccc-e878-412b-a43d-b8b0885f7523): select count(*) from iotdatademo
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20171204222828_f697cccc-e878-412b-a43d-b8b0885f7523); Time taken: 1.373 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20171204222828_f697cccc-e878-412b-a43d-b8b0885f7523): select count(*) from iotdatademo
INFO  : Query ID = hive_20171204222828_f697cccc-e878-412b-a43d-b8b0885f7523
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : In order to change the average load for a reducer (in bytes):
INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
INFO  : In order to limit the maximum number of reducers:
INFO  :   set hive.exec.reducers.max=<number>
INFO  : In order to set a constant number of reducers:
INFO  :   set mapreduce.job.reduces=<number>
INFO  : Starting Spark Job = 2f4f87bc-df31-43f5-aafd-e4e3af6db389
INFO  : Running with YARN Application = application_1512397511699_0001
INFO  : Kill Command = /usr/lib/hadoop/bin/yarn application -kill application_1512397511699_0001
INFO  :
Query Hive on Spark job[0] stages:
INFO  : 0
INFO  : 1
INFO  :
Status: Running (Hive on Spark job[0])
INFO  : Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
INFO  : 2017-12-04 22:28:33,985 Stage-0_0: 0(+1)/1      Stage-1_0: 0/1
INFO  : 2017-12-04 22:28:37,044 Stage-0_0: 0(+1)/1      Stage-1_0: 0/1
INFO  : 2017-12-04 22:28:38,057 Stage-0_0: 1/1 Finished Stage-1_0: 0(+1)/1
INFO  : 2017-12-04 22:28:39,065 Stage-0_0: 1/1 Finished Stage-1_0: 1/1 Finished
INFO  : Status: Finished successfully in 14.16 seconds
INFO  : =====Spark Job[2f4f87bc-df31-43f5-aafd-e4e3af6db389] statistics=====
INFO  : HIVE
INFO  :         CREATED_FILES: 1
INFO  :         DESERIALIZE_ERRORS: 0
INFO  :         RECORDS_OUT_INTERMEDIATE: 1
INFO  :         RECORDS_IN: 864010
INFO  :         RECORDS_OUT_0: 1
INFO  : Spark Job[2f4f87bc-df31-43f5-aafd-e4e3af6db389] Metrics
INFO  :         ExecutorDeserializeTime: 1904
INFO  :         ExecutorRunTime: 2769
INFO  :         ResultSize: 3528
INFO  :         JvmGCTime: 111
INFO  :         ResultSerializationTime: 0
INFO  :         MemoryBytesSpilled: 0
INFO  :         DiskBytesSpilled: 0
INFO  :         BytesRead: 46841465
INFO  :         RemoteBlocksFetched: 0
INFO  :         LocalBlocksFetched: 1
INFO  :         TotalBlocksFetched: 1
INFO  :         FetchWaitTime: 0
INFO  :         RemoteBytesRead: 0
INFO  :         ShuffleBytesWritten: 42
INFO  :         ShuffleWriteTime: 6398688
INFO  : Execution completed successfully
INFO  : Completed executing command(queryId=hive_20171204222828_f697cccc-e878-412b-a43d-b8b0885f7523); Time taken: 30.155 seconds
INFO  : OK
+---------+--+
|   _c0   |
+---------+--+
| 864010  |
+---------+--+
1 row selected (31.798 seconds)
0: jdbc:hive2://cdh-vm:10000> select count(*) from iotdatademo;
INFO  : Compiling command(queryId=hive_20171204222828_77800344-d77d-4d19-bc79-0cdcf7cc90bc): select count(*) from iotdatademo
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20171204222828_77800344-d77d-4d19-bc79-0cdcf7cc90bc); Time taken: 0.114 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20171204222828_77800344-d77d-4d19-bc79-0cdcf7cc90bc): select count(*) from iotdatademo
INFO  : Query ID = hive_20171204222828_77800344-d77d-4d19-bc79-0cdcf7cc90bc
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : In order to change the average load for a reducer (in bytes):
INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
INFO  : In order to limit the maximum number of reducers:
INFO  :   set hive.exec.reducers.max=<number>
INFO  : In order to set a constant number of reducers:
INFO  :   set mapreduce.job.reduces=<number>
INFO  : Starting Spark Job = 58ba32e1-4650-4e7c-9a0a-1f4cbfb0aa07
INFO  : Running with YARN Application = application_1512397511699_0001
INFO  : Kill Command = /usr/lib/hadoop/bin/yarn application -kill application_1512397511699_0001
INFO  :
Query Hive on Spark job[1] stages:
INFO  : 2
INFO  : 3
INFO  :
Status: Running (Hive on Spark job[1])
INFO  : Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
INFO  : 2017-12-04 22:28:50,432 Stage-2_0: 1/1 Finished Stage-3_0: 0(+1)/1
INFO  : 2017-12-04 22:28:51,441 Stage-2_0: 1/1 Finished Stage-3_0: 1/1 Finished
INFO  : Status: Finished successfully in 3.03 seconds
INFO  : =====Spark Job[58ba32e1-4650-4e7c-9a0a-1f4cbfb0aa07] statistics=====
INFO  : Spark Job[58ba32e1-4650-4e7c-9a0a-1f4cbfb0aa07] Metrics
INFO  :         ExecutorDeserializeTime: 71
INFO  :         ExecutorRunTime: 1172
INFO  :         ResultSize: 3528
INFO  :         JvmGCTime: 16
INFO  :         ResultSerializationTime: 0
INFO  :         MemoryBytesSpilled: 0
INFO  :         DiskBytesSpilled: 0
INFO  :         BytesRead: 46842211
INFO  :         RemoteBlocksFetched: 0
INFO  :         LocalBlocksFetched: 1
INFO  :         TotalBlocksFetched: 1
INFO  :         FetchWaitTime: 0
INFO  :         RemoteBytesRead: 0
INFO  :         ShuffleBytesWritten: 42
INFO  :         ShuffleWriteTime: 845403
INFO  : HIVE
INFO  :         CREATED_FILES: 1
INFO  :         DESERIALIZE_ERRORS: 0
INFO  :         RECORDS_OUT_INTERMEDIATE: 1
INFO  :         RECORDS_IN: 864010
INFO  :         RECORDS_OUT_0: 1
INFO  : Execution completed successfully
INFO  : Completed executing command(queryId=hive_20171204222828_77800344-d77d-4d19-bc79-0cdcf7cc90bc); Time taken: 3.124 seconds
INFO  : OK
+---------+--+
|   _c0   |
+---------+--+
| 864010  |
+---------+--+
1 row selected (3.284 seconds)
0: jdbc:hive2://cdh-vm:10000> select eday, avg(speed) from iotdatademo group by eday order by eday;
INFO  : Compiling command(queryId=hive_20171204222929_66b80ffa-4d62-4dd5-ac70-b9707864d4c0): select eday, avg(speed) from iotdatademo group by eday order by eday
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:eday, type:bigint, comment:null), FieldSchema(name:_c1, type:double, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20171204222929_66b80ffa-4d62-4dd5-ac70-b9707864d4c0); Time taken: 0.21 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20171204222929_66b80ffa-4d62-4dd5-ac70-b9707864d4c0): select eday, avg(speed) from iotdatademo group by eday order by eday
INFO  : Query ID = hive_20171204222929_66b80ffa-4d62-4dd5-ac70-b9707864d4c0
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : In order to change the average load for a reducer (in bytes):
INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
INFO  : In order to limit the maximum number of reducers:
INFO  :   set hive.exec.reducers.max=<number>
INFO  : In order to set a constant number of reducers:
INFO  :   set mapreduce.job.reduces=<number>
INFO  : Starting Spark Job = d1a63e33-46ff-4540-9b77-b64070b74256
INFO  : Running with YARN Application = application_1512397511699_0001
INFO  : Kill Command = /usr/lib/hadoop/bin/yarn application -kill application_1512397511699_0001
INFO  :
Query Hive on Spark job[3] stages:
INFO  : 6
INFO  : 7
INFO  : 8
INFO  :
Status: Running (Hive on Spark job[3])
INFO  : Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
INFO  : 2017-12-04 22:29:18,756 Stage-6_0: 0/1  Stage-7_0: 0/2  Stage-8_0: 0/1
INFO  : 2017-12-04 22:29:19,780 Stage-6_0: 0(+1)/1      Stage-7_0: 0/2  Stage-8_0: 0/1
INFO  : 2017-12-04 22:29:20,786 Stage-6_0: 1/1 Finished Stage-7_0: 2/2 Finished Stage-8_0: 1/1 Finished
INFO  : Status: Finished successfully in 3.06 seconds
INFO  : =====Spark Job[d1a63e33-46ff-4540-9b77-b64070b74256] statistics=====
INFO  : HIVE
INFO  :         CREATED_FILES: 1
INFO  :         DESERIALIZE_ERRORS: 0
INFO  :         RECORDS_OUT_INTERMEDIATE: 14
INFO  :         RECORDS_IN: 864010
INFO  :         RECORDS_OUT_0: 7
INFO  : Spark Job[d1a63e33-46ff-4540-9b77-b64070b74256] Metrics
INFO  :         ExecutorDeserializeTime: 246
INFO  :         ExecutorRunTime: 1864
INFO  :         ResultSize: 6350
INFO  :         JvmGCTime: 52
INFO  :         ResultSerializationTime: 0
INFO  :         MemoryBytesSpilled: 0
INFO  :         DiskBytesSpilled: 0
INFO  :         BytesRead: 46842469
INFO  :         RemoteBlocksFetched: 0
INFO  :         LocalBlocksFetched: 4
INFO  :         TotalBlocksFetched: 4
INFO  :         FetchWaitTime: 0
INFO  :         RemoteBytesRead: 0
INFO  :         ShuffleBytesWritten: 501
INFO  :         ShuffleWriteTime: 39846606
INFO  : Execution completed successfully
INFO  : Completed executing command(queryId=hive_20171204222929_66b80ffa-4d62-4dd5-ac70-b9707864d4c0); Time taken: 3.137 seconds
INFO  : OK
+-------+--------------------+--+
| eday  |        _c1         |
+-------+--------------------+--+
| 0     | 80.42217204861018  |
| 1     | 80.4242077305864   |
| 2     | 80.516892013888    |
| 3     | 80.42997673611163  |
| 4     | 80.62740798611237  |
| 5     | 80.49621712963015  |
| 6     | 80.54539832175986  |
+-------+--------------------+--+
7 rows selected (3.407 seconds)
0: jdbc:hive2://cdh-vm:10000> !exit
Closing: 0: jdbc:hive2://cdh-vm:10000