Submit Spark Job Example

donghua@ubuntu:~$ ./spark-1.6.1/sbin/start-master.sh

starting org.apache.spark.deploy.master.Master, logging to /home/donghua/spark-1.6.1/logs/spark-donghua-org.apache.spark.deploy.master.Master-1-ubuntu.out

donghua@ubuntu:~$ tail -f /home/donghua/spark-1.6.1/logs/spark-donghua-org.apache.spark.deploy.master.Master-1-ubuntu.out

16/11/07 22:01:19 INFO SecurityManager: Changing modify acls to: donghua

16/11/07 22:01:19 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(donghua); users with modify permissions: Set(donghua)

16/11/07 22:01:20 INFO Utils: Successfully started service 'sparkMaster' on port 7077.

16/11/07 22:01:20 INFO Master: Starting Spark master at spark://ubuntu:7077

16/11/07 22:01:20 INFO Master: Running Spark version 1.6.1

16/11/07 22:01:20 INFO Utils: Successfully started service 'MasterUI' on port 8080.

16/11/07 22:01:20 INFO MasterWebUI: Started MasterWebUI at http://192.168.6.152:8080

16/11/07 22:01:20 INFO Utils: Successfully started service on port 6066.

16/11/07 22:01:20 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066

16/11/07 22:01:21 INFO Master: I have been elected leader! New state: ALIVE

donghua@ubuntu:~$ ./spark-1.6.1/sbin/start-slave.sh spark://127.0.1.1:7077

starting org.apache.spark.deploy.worker.Worker, logging to /home/donghua/spark-1.6.1/logs/spark-donghua-org.apache.spark.deploy.worker.Worker-1-ubuntu.out

donghua@ubuntu:~$

donghua@ubuntu:~$ tail -f /home/donghua/spark-1.6.1/logs/spark-donghua-org.apache.spark.deploy.worker.Worker-1-ubuntu.out

16/11/07 22:08:21 INFO SecurityManager: Changing modify acls to: donghua

16/11/07 22:08:21 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(donghua); users with modify permissions: Set(donghua)

16/11/07 22:08:22 INFO Utils: Successfully started service 'sparkWorker' on port 43925.

16/11/07 22:08:22 INFO Worker: Starting Spark worker 192.168.6.152:43925 with 1 cores, 2.8 GB RAM

16/11/07 22:08:22 INFO Worker: Running Spark version 1.6.1

16/11/07 22:08:22 INFO Worker: Spark home: /home/donghua/spark-1.6.1

16/11/07 22:08:23 INFO Utils: Successfully started service 'WorkerUI' on port 8081.

16/11/07 22:08:23 INFO WorkerWebUI: Started WorkerWebUI at http://192.168.6.152:8081

16/11/07 22:08:23 INFO Worker: Connecting to master 127.0.1.1:7077...

16/11/07 22:08:23 INFO Worker: Successfully registered with master spark://ubuntu:7077

donghua@ubuntu:~$ spark-1.6.1/bin/spark-submit --class spacewalk.SpaceWalk --master spark://127.0.1.1:7077 MongoDB_Spark_Course/build/libs/SparkCourse-1.0.0-SNAPSHOT.jar

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

16/11/07 22:18:45 INFO SparkContext: Running Spark version 1.6.1

16/11/07 22:18:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/11/07 22:18:46 WARN Utils: Your hostname, ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.6.152 instead (on interface ens33)

16/11/07 22:18:46 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

16/11/07 22:18:46 INFO SecurityManager: Changing view acls to: donghua

16/11/07 22:18:46 INFO SecurityManager: Changing modify acls to: donghua

16/11/07 22:18:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(donghua); users with modify permissions: Set(donghua)

16/11/07 22:18:47 INFO Utils: Successfully started service 'sparkDriver' on port 40563.

16/11/07 22:18:47 INFO Slf4jLogger: Slf4jLogger started

16/11/07 22:18:47 INFO Remoting: Starting remoting

16/11/07 22:18:48 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.6.152:43077]

16/11/07 22:18:48 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 43077.

16/11/07 22:18:48 INFO SparkEnv: Registering MapOutputTracker

16/11/07 22:18:48 INFO SparkEnv: Registering BlockManagerMaster

16/11/07 22:18:48 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-5c316a65-2114-4c0f-bc34-f13f9899296d

16/11/07 22:18:48 INFO MemoryStore: MemoryStore started with capacity 517.4 MB

16/11/07 22:18:48 INFO SparkEnv: Registering OutputCommitCoordinator

16/11/07 22:18:48 INFO Utils: Successfully started service 'SparkUI' on port 4040.

16/11/07 22:18:48 INFO SparkUI: Started SparkUI at http://192.168.6.152:4040

16/11/07 22:18:48 INFO HttpFileServer: HTTP File server directory is /tmp/spark-67642081-2f62-481e-ae5c-a86928dd76bf/httpd-175bafdb-25ae-4ef0-9afb-887d9152bf80

16/11/07 22:18:48 INFO HttpServer: Starting HTTP Server

16/11/07 22:18:48 INFO Utils: Successfully started service 'HTTP file server' on port 34672.

16/11/07 22:18:49 INFO SparkContext: Added JAR file:/home/donghua/MongoDB_Spark_Course/build/libs/SparkCourse-1.0.0-SNAPSHOT.jar at http://192.168.6.152:34672/jars/SparkCourse-1.0.0-SNAPSHOT.jar with timestamp 1478528329476

16/11/07 22:18:49 INFO Executor: Starting executor ID driver on host localhost

16/11/07 22:18:49 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39169.

16/11/07 22:18:49 INFO NettyBlockTransferService: Server created on 39169

16/11/07 22:18:49 INFO BlockManagerMaster: Trying to register BlockManager

16/11/07 22:18:49 INFO BlockManagerMasterEndpoint: Registering block manager localhost:39169 with 517.4 MB RAM, BlockManagerId(driver, localhost, 39169)

16/11/07 22:18:49 INFO BlockManagerMaster: Registered BlockManager

16/11/07 22:18:50 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 200.0 B, free 200.0 B)

16/11/07 22:18:50 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 377.0 B, free 577.0 B)

16/11/07 22:18:50 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:39169 (size: 377.0 B, free: 517.4 MB)

16/11/07 22:18:50 INFO SparkContext: Created broadcast 0 from broadcast at MongoRDD.scala:145

16/11/07 22:18:51 INFO cluster: Cluster created with settings {hosts=[127.0.0.1:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}

16/11/07 22:18:51 INFO cluster: Adding discovered server 127.0.0.1:27017 to client view of cluster

16/11/07 22:18:51 INFO cluster: Cluster description not yet available. Waiting for 30000 ms before timing out

16/11/07 22:18:51 INFO connection: Opened connection [connectionId{localValue:1, serverValue:15}] to 127.0.0.1:27017

16/11/07 22:18:51 INFO cluster: Monitor thread successfully connected to server with description ServerDescription{address=127.0.0.1:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 10]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=3238620}

16/11/07 22:18:51 INFO cluster: Discovered cluster type of STANDALONE

16/11/07 22:18:51 INFO MongoClientCache: Creating MongoClient: [127.0.0.1:27017]

16/11/07 22:18:51 INFO connection: Opened connection [connectionId{localValue:2, serverValue:16}] to 127.0.0.1:27017

16/11/07 22:18:51 INFO MongoSplitVectorPartitioner: No splitKeys were calculated by the splitVector command, proceeding with a single partition. If this is undesirable try lowering 'maxChunkSize' to produce more partitions.

16/11/07 22:18:51 INFO SparkContext: Starting job: foreach at SpaceWalk.java:103

16/11/07 22:18:51 INFO DAGScheduler: Registering RDD 1 (flatMapToPair at SpaceWalk.java:47)

16/11/07 22:18:51 INFO DAGScheduler: Got job 0 (foreach at SpaceWalk.java:103) with 1 output partitions

16/11/07 22:18:51 INFO DAGScheduler: Final stage: ResultStage 1 (foreach at SpaceWalk.java:103)

16/11/07 22:18:51 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)

16/11/07 22:18:51 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)

16/11/07 22:18:51 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[1] at flatMapToPair at SpaceWalk.java:47), which has no missing parents

16/11/07 22:18:51 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 6.1 KB, free 6.7 KB)

16/11/07 22:18:51 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.0 KB, free 9.7 KB)

16/11/07 22:18:51 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:39169 (size: 3.0 KB, free: 517.4 MB)

16/11/07 22:18:51 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006

16/11/07 22:18:51 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[1] at flatMapToPair at SpaceWalk.java:47)

16/11/07 22:18:51 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks

16/11/07 22:18:51 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,ANY, 2295 bytes)

16/11/07 22:18:51 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)

16/11/07 22:18:51 INFO Executor: Fetching http://192.168.6.152:34672/jars/SparkCourse-1.0.0-SNAPSHOT.jar with timestamp 1478528329476

16/11/07 22:18:52 INFO Utils: Fetching http://192.168.6.152:34672/jars/SparkCourse-1.0.0-SNAPSHOT.jar to /tmp/spark-67642081-2f62-481e-ae5c-a86928dd76bf/userFiles-a63f8515-d9df-494d-a07e-7654142bcff7/fetchFileTemp3583183982411046383.tmp

16/11/07 22:18:52 INFO Executor: Adding file:/tmp/spark-67642081-2f62-481e-ae5c-a86928dd76bf/userFiles-a63f8515-d9df-494d-a07e-7654142bcff7/SparkCourse-1.0.0-SNAPSHOT.jar to class loader

16/11/07 22:18:54 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1158 bytes result sent to driver

16/11/07 22:18:54 INFO DAGScheduler: ShuffleMapStage 0 (flatMapToPair at SpaceWalk.java:47) finished in 2.252 s

16/11/07 22:18:54 INFO DAGScheduler: looking for newly runnable stages

16/11/07 22:18:54 INFO DAGScheduler: running: Set()

16/11/07 22:18:54 INFO DAGScheduler: waiting: Set(ResultStage 1)

16/11/07 22:18:54 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 2231 ms on localhost (1/1)

16/11/07 22:18:54 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool

16/11/07 22:18:54 INFO DAGScheduler: failed: Set()

16/11/07 22:18:54 INFO DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[2] at reduceByKey at SpaceWalk.java:80), which has no missing parents

16/11/07 22:18:54 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.9 KB, free 12.6 KB)

16/11/07 22:18:54 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1780.0 B, free 14.3 KB)

16/11/07 22:18:54 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:39169 (size: 1780.0 B, free: 517.4 MB)

16/11/07 22:18:54 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006

16/11/07 22:18:54 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (ShuffledRDD[2] at reduceByKey at SpaceWalk.java:80)

16/11/07 22:18:54 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks

16/11/07 22:18:54 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, partition 0,NODE_LOCAL, 1966 bytes)

16/11/07 22:18:54 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)

16/11/07 22:18:54 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks

16/11/07 22:18:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms

Ken Bowersox 797

Don Peterson 257

Vladimir Dzhanibekov 512

Mike Collins 89

Claude Nicollier 490

Alexandr Ivanchenkov 125

Woody Spring 746

Ron Garan 1623

Vladimir Lyakhov 427

Patrick Forrester 705

Harrison Schmidt 1333

Gennady Strekalov 1322

Yuri Lonchakov 627

Bill Pogue 814

Valentin Lebedev 153

Hans Schlegel 405

Gennady Padalka 1601

Louc Chretien 357

Mikhail Kornienko 402

Dave Leestma 209

Heide Stefanyshyn 2022

Stan Love 923

Mike Massimino 1844

Joe Kerwin 245

Oleg Skripochka 1001

Victor Savinykh 298

Dave Wolf 2516

Oleg Kotov 1006

Joe Allen 710

Rich Clifford 363

Greg Chamitoff 823

Musa Manarov 2058

Nicole Stott 395

Anatoli Solovyov 1753

Nick Patrick 1093

Anton Shkaplerov 375

Alexandr Serebrov 1913

Alan Shepard 567

Dan Burbank 431

Alexandr Balandin 652

Jim Voss 1365

Jeff Wisoff 1193

Jack Lousma 662

Sergei Zaletin 330

Valeri Tsibliev 297

Mike Fincke 2916

Sunita Williams 3040

John Grunsfeld 3510

Alexandr Poleshchuk 597

James Irwin 1120

Pavel Vinogradov 2419

Richard Gordon 174

Thomas Reiter 856

Danny Olivas 2068

Linda Godwin 615

Anatoly Solovyev 3086

Oleg Kononenko 1108

Tammy Jernigan 475

Andrew Feustel 2538

Mike Good 958

Mike Gernhardt 1396

Alvin Drew 768

Leroy Chiao 2176

Alexei Leonov 12

Rick Mastracchio 2311

Alexandr Viktorenko 1184

Dave Williams 1067

Chris Cassidy 1875

Kathy Thornton 1270

Yuri Onufrenko 1830

Robert Satcher 739

Richard Arnold 754

Alexander Kaleri 1318

Leonid Kizim 1889

Christer Fugelsang 820

Alexander Samokutyaev 383

Doug Wheelock 1241

Aleksei Yeliseyov 37

Vladimir Solovyov 1889

Russ Schweickart 51

Yuri Usachev 1849

Ed White 36

Roman Romanenko 398

Bill McArthur 1462

Alexandr Volkov 610

Steve Smith 2988

Christer Fuglesang 1094

Jim Newman 2593

Story Musgrave 1579

Frank Culbertson 304

Dmitry Kondratiev 614

Bob Behnken 1093

Tom Akers 1270

Alexandr Kaleri 123

Neil Armstrong 157

Vladimir Kovalyonok 125

Robert Curbeam 1545

Don Pettit 797

Allen Bean 613

Mikhail Tyurin 1532

Sergei Avdeyev 2518

Tim Kopra 332

Victor Afanasyev 2314

Lee Morin 847

Rick Hieb 553

Valery Korzun 1339

Yuri Romanenko 615

Tom Marshburn 1469

Bruce McCandless 725

Mario Runco 268

Victor Lazerov 0

Bill Lenoir 0

Greg Harbaugh 1109

Eugene Cernan 1460

Peggy Whitson 2386

Valeri Tsibliyev 852

Dan Barry 1549

Yuri Onufrienko 359

John Herrington 1195

Anatoli Berezovsky 153

Bob Curbeam 1189

Bernard Harris 279

Pierre Heignere 379

Jerry Ross 3501

Alexandr Laveykin 527

Gennady Manakov 782

Yri Onufrienko 363

Clay Anderson 1091

Alexandr Alexandrov 344

Ed Mitchell 567

Jay Apt 626

Ed Gibson 921

Alexander Misurkin 1201

Mike Lopez 3344

Dave Griggs 190

Yuri Malenchenko 1436

Shane Kimbrough 772

Anatoli Artsebarsky 1937

Paul Weitz 136

Andy Thomas 381

Mike Foale 1364

Jim Reilly 1843

Philippe Perrin 1170

John Phillips 298

Steve Bowen 2838

Mark Lee 1561

Kathy Sullivan 209

Doug Wheellock 1369

Susan Helms 536

Buzz Aldrin 487

Sergei Treschev 321

Nikola Budarin 2672

Robert Behnken 1159

Jerry Linenger 297

David Scott 1200

Mike Fossum 2912

Jeff Williams 1149

Winston Scott 1175

Talgat Musabeyev 2478

Vladimir Dezhurov 2257

Sergei Krikalev 2488

Svetlana Savitskaya 214

Michael Good 835

Randy Bresnick 710

Joseph Acaba 777

Yevgeni Khrunov 37

John Young 1218

Tom Jones 1189

Dan Bursch 708

Bill Fisher 698

Dale Gardner 710

Carl Walz 1137

Steve MacLean 431

Takao Doi 762

Valeri Tokarev 666

Paul Richards 381

Owen Garriott 823

Valeri Ryumin 83

Salizhan Sharipov 598

Al Worden 39

Mike Foreman 1939

Clayton Anderson 1218

Oleg Makarov 0

Akihiko Hoshide 1283

Mike Barratt 306

Scott Parazynski 2825

Fyodor Yurchikhin 2324

Carl Meade 411

Dan Tani 2351

James Van 1296

Pete Conrad 753

Carlos Noriega 1160

Rex Walheim 2183

George Nelson 598

Maxim Suraev 344

Georgi Grechko 88

Charles Duke 1218

Jeff Hoffman 1512

Chris Hadfield 890

Steve Robinson 1205

Yuri Gidzenko 218

Tom Mattingly 83

Yuri Malenchecko 374

Michael Lopez 716

Gerald Carr 949

Pierre Thuot 553

Ron Evans 67

Edward Lu 374

Bob Stewart 725

Joe Tanner 2789

Pat Forrester 825

Piers Sellers 2470

Soichi Noguchi 1205

Vladimir Titov 1113

Tracy Caldwell 1369

David Low 350

Steve Swanson 1582

Fyodor Yurchikin 789

Sergei Volkov 1116

Garrett Reisman 1272

Luca Parmitano 459

Rick Linnehan 2531

16/11/07 22:18:54 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1165 bytes result sent to driver

16/11/07 22:18:54 INFO DAGScheduler: ResultStage 1 (foreach at SpaceWalk.java:103) finished in 0.194 s

16/11/07 22:18:54 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 205 ms on localhost (1/1)

16/11/07 22:18:54 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool

16/11/07 22:18:54 INFO DAGScheduler: Job 0 finished: foreach at SpaceWalk.java:103, took 2.825976 s

16/11/07 22:18:54 INFO SparkContext: Starting job: foreachPartition at DocumentRDDFunctions.scala:50

16/11/07 22:18:54 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 143 bytes

16/11/07 22:18:54 INFO DAGScheduler: Got job 1 (foreachPartition at DocumentRDDFunctions.scala:50) with 1 output partitions

16/11/07 22:18:54 INFO DAGScheduler: Final stage: ResultStage 3 (foreachPartition at DocumentRDDFunctions.scala:50)

16/11/07 22:18:54 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)

16/11/07 22:18:54 INFO DAGScheduler: Missing parents: List()

16/11/07 22:18:54 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[3] at flatMap at SpaceWalk.java:101), which has no missing parents

16/11/07 22:18:54 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 4.6 KB, free 19.0 KB)

16/11/07 22:18:54 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.5 KB, free 21.5 KB)

16/11/07 22:18:54 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:39169 (size: 2.5 KB, free: 517.4 MB)

16/11/07 22:18:54 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006

16/11/07 22:18:54 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[3] at flatMap at SpaceWalk.java:101)

16/11/07 22:18:54 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks

16/11/07 22:18:54 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 2, localhost, partition 0,NODE_LOCAL, 1966 bytes)

16/11/07 22:18:54 INFO Executor: Running task 0.0 in stage 3.0 (TID 2)

16/11/07 22:18:54 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks

16/11/07 22:18:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms

16/11/07 22:18:54 INFO cluster: Cluster created with settings {hosts=[127.0.0.1:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}

16/11/07 22:18:54 INFO cluster: Adding discovered server 127.0.0.1:27017 to client view of cluster

16/11/07 22:18:54 INFO connection: Opened connection [connectionId{localValue:3, serverValue:17}] to 127.0.0.1:27017

16/11/07 22:18:54 INFO cluster: Cluster description not yet available. Waiting for 30000 ms before timing out

16/11/07 22:18:54 INFO cluster: Monitor thread successfully connected to server with description ServerDescription{address=127.0.0.1:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 10]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=956829}

16/11/07 22:18:54 INFO cluster: Discovered cluster type of STANDALONE

16/11/07 22:18:54 INFO MongoClientCache: Creating MongoClient: [127.0.0.1:27017]

16/11/07 22:18:54 INFO connection: Opened connection [connectionId{localValue:4, serverValue:18}] to 127.0.0.1:27017

16/11/07 22:18:54 INFO Executor: Finished task 0.0 in stage 3.0 (TID 2). 1165 bytes result sent to driver

16/11/07 22:18:54 INFO DAGScheduler: ResultStage 3 (foreachPartition at DocumentRDDFunctions.scala:50) finished in 0.149 s

16/11/07 22:18:54 INFO DAGScheduler: Job 1 finished: foreachPartition at DocumentRDDFunctions.scala:50, took 0.188457 s

16/11/07 22:18:54 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 2) in 152 ms on localhost (1/1)

16/11/07 22:18:54 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool

16/11/07 22:18:54 INFO MongoClientCache: Closing MongoClient: [127.0.0.1:27017]

16/11/07 22:18:54 INFO connection: Closed connection [connectionId{localValue:2, serverValue:16}] to 127.0.0.1:27017 because the pool has been closed.

16/11/07 22:18:54 INFO SparkContext: Invoking stop() from shutdown hook

16/11/07 22:18:54 INFO MongoClientCache: Closing MongoClient: [127.0.0.1:27017]

16/11/07 22:18:54 INFO connection: Closed connection [connectionId{localValue:4, serverValue:18}] to 127.0.0.1:27017 because the pool has been closed.

16/11/07 22:18:54 INFO SparkUI: Stopped Spark web UI at http://192.168.6.152:4040

16/11/07 22:18:54 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

16/11/07 22:18:54 INFO MemoryStore: MemoryStore cleared

16/11/07 22:18:54 INFO BlockManager: BlockManager stopped

16/11/07 22:18:54 INFO BlockManagerMaster: BlockManagerMaster stopped

16/11/07 22:18:54 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

16/11/07 22:18:54 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.

16/11/07 22:18:54 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.

16/11/07 22:18:54 INFO SparkContext: Successfully stopped SparkContext

16/11/07 22:18:54 INFO ShutdownHookManager: Shutdown hook called

16/11/07 22:18:54 INFO ShutdownHookManager: Deleting directory /tmp/spark-67642081-2f62-481e-ae5c-a86928dd76bf/httpd-175bafdb-25ae-4ef0-9afb-887d9152bf80

16/11/07 22:18:54 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.

16/11/07 22:18:54 INFO ShutdownHookManager: Deleting directory /tmp/spark-67642081-2f62-481e-ae5c-a86928dd76bf

donghua@ubuntu:~$