Wednesday, January 3, 2018

Using Open Source R-Studio Server connecting to Kerberos-enabled Hadoop

Step 1: Add line "SPARK_HOME=${SPARK_HOME-'/opt/cloudera/parcels/CDH/lib/spark/'}" to end of file "/usr/lib64/R/etc/Renviron"

Step 2: Connect to Spark using sparklyr inside R-Studio Server

> install.packages("sparklyr")
> library(sparklyr)
> readRenviron("/usr/lib64/R/etc/Renviron")
> sc <- spark_connect(master = "yarn-client",version = "1.6.0", config = list
(default = list(spark.yarn.keytab = "/home/donghua/donghua.keytab", spark.yarn.principal = "donghua@DBAGLOBE.COM")))
> sc
$master
[1] "yarn-client"

$method
[1] "shell"

$app_name
[1] "sparklyr"

$config
$config$default
$config$default$spark.yarn.keytab
[1] "/home/donghua/donghua.keytab"

$config$default$spark.yarn.principal
[1] "donghua@DBAGLOBE.COM"



$spark_home
[1] "/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/spark"

$backend
A connection with                               
description "->localhost:46015"
class       "sockconn"         
mode        "wb"               
text        "binary"           
opened      "opened"           
can read    "yes"              
can write   "yes"              

$monitor
A connection with                              
description "->localhost:8880"
class       "sockconn"        
mode        "rb"              
text        "binary"          
opened      "opened"          
can read    "yes"             
can write   "yes"             

$output_file
[1] "/tmp/RtmpXWaXfE/file7af1ca61a03_spark.log"

$spark_context
<jobj[6]>
  class org.apache.spark.SparkContext
  org.apache.spark.SparkContext@355d7d99

$java_context
<jobj[7]>
  class org.apache.spark.api.java.JavaSparkContext
  org.apache.spark.api.java.JavaSparkContext@ef616c5

attr(,"class")
[1] "spark_connection"       "spark_shell_connection" "DBIConnection"   


image

> library(DBI)
> iotdatademo <- dbGetQuery(sc, 'Select * from default.iotdatademo limit 10')
> iotdatademo

image

Reference URL: https://medium.com/@bkvarda/sparklyr-r-interface-for-spark-and-kerberos-on-cloudera-80abf5f6b4ad

35 comments:

  1. To inquire about the enormous measure of information we need an expert and all around experienced Data Scientist, to end up one of the specialists among the Data Scientist swarm, you should upgrade your range of abilities in huge information in a propelled manner. ExcelR Data Science Courses

    ReplyDelete
  2. I am genuinely thankful to the holder of this web page who has shared this wonderful paragraph at at this place
    360digitmg

    ReplyDelete
  3. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.Best data science courses in hyerabad

    ReplyDelete
  4. This is a great post. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again. photographe

    ReplyDelete
  5. I am very happy to discover your post as it will become on top in my collection of favorite blogs to visit. WordPress Developer Brisbane

    ReplyDelete
  6. Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include.
    토토사이트

    ReplyDelete
  7. I like this post,And I guess that they having fun to read this post,they shall take a good site to make a information,thanks for sharing it to me.
    파워볼사이트

    ReplyDelete
  8. Superbly written article, if only all bloggers offered the same content as you, the internet would be a far better place 스포츠토토사이트

    ReplyDelete
  9. The registered guarantor at reliable. Always fighting! I'll come often. Great work! 보증업체

    ReplyDelete
  10. I can grow up with. Please continue to do well and always do well 먹튀폴리스 I'm surprised there's such a wonderful article

    ReplyDelete
  11. I live in a different country than you This post is invaluable 카지노커뮤니티

    ReplyDelete
  12. "It’s amazing in support of me to have a site, which is good in favor of my know-how. thanks admin" 메이저놀이터

    ReplyDelete
  13. understand everything, this is really fun and enjoyable for me. Thank you guys for sharing this. Click here, 토토사이트검증

    ReplyDelete
  14. Good website! I truly love how it is easy on my eyes it is. I am wondering how I might be notified whenever a new post has been made. I have subscribed to your RSS which may do the trick? Have a great day! 먹튀사이트

    ReplyDelete
  15. Great post, you have pointed out some fantastic points , I likewise think this s a very wonderful website. i99pro

    ReplyDelete
  16. Great articles and great layout. Your blog post deserves all of the positive feedback it’s been getting. 안전놀이터

    ReplyDelete
  17. You have done a great job on this article. It’s very readable and highly intelligent. You have even managed to make it understandable and easy to read. You have some real writing talent. Thank you. cheap hosting in indiaonohosting

    ReplyDelete
  18. An fascinating discussion is value comment. I think that it is best to write extra on this matter, it won’t be a taboo topic however generally people are not enough to talk on such topics. To the next. Cheers 안전놀이터

    ReplyDelete
  19. Great post, you have pointed out some fantastic points , I likewise think this s a very wonderful website. บาคาร่า

    ReplyDelete
  20. I am thankful to you for sharing this plethora of useful information. I found this resource utmost beneficial for me. Thanks a lot for hard work. 먹튀검증

    ReplyDelete
  21. Thank you so much as you have been willing to share information with us. We will forever admire all you have done here because you have made my work as easy as ABC. best digital perm

    ReplyDelete
  22. I cannot wait to dig deep and kickoff utilizing resources that I received from you. Your exuberance is refreshing. เว็บพนันบอลดีที่สุด

    ReplyDelete
  23. Everything has its value. Thanks for sharing this informative information with us. GOOD works! here

    ReplyDelete
  24. It is my first visit to your blog, and I am very impressed with the articles that you serve. Give adequate knowledge for me. Thank you for sharing useful material. I will be back for the more great post. 안전놀이터

    ReplyDelete
  25. Good website! I truly love how it is easy on my eyes it is. I am wondering how I might be notified whenever a new post has been made. I have subscribed to your RSS which may do the trick? Have a great day! movie4me

    ReplyDelete
  26. Pretty good post. I have just stumbled upon your blog and enjoyed reading your blog posts very much. I am looking for new posts to get more precious info. Big thanks for the useful info. subcene

    ReplyDelete
  27. I really like your writing style, great information, thankyou for posting. matka tips

    ReplyDelete
  28. Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work. matka

    ReplyDelete
  29. Great post, you have pointed out some fantastic points , I likewise think this s a very wonderful website. Watergardens at Canberra

    ReplyDelete
  30. Wonderful blog! I found it while surfing around on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News? I’ve been trying for a while but I never seem to get there! Appreciate it. Pollen Collection Price

    ReplyDelete
  31. Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work. 슬롯사이트

    ReplyDelete
  32. Hi there! Nice stuff, do keep me posted when you post again something like this! bolly4u

    ReplyDelete
  33. That is the excellent mindset, nonetheless is just not help to make every sence whatsoever preaching about that mather. Virtually any method many thanks in addition to i had endeavor to promote your own article in to delicius nevertheless it is apparently a dilemma using your information sites can you please recheck the idea. thanks once more. 안전놀이터

    ReplyDelete
  34. Thank you for such a well written article. It’s full of insightful information and entertaining descriptions. Your point of view is the best among many. 먹튀검증

    ReplyDelete