Wednesday, January 3, 2018

Using Open Source R-Studio Server connecting to Kerberos-enabled Hadoop

Step 1: Add line "SPARK_HOME=${SPARK_HOME-'/opt/cloudera/parcels/CDH/lib/spark/'}" to end of file "/usr/lib64/R/etc/Renviron"

Step 2: Connect to Spark using sparklyr inside R-Studio Server

> install.packages("sparklyr")
> library(sparklyr)
> readRenviron("/usr/lib64/R/etc/Renviron")
> sc <- spark_connect(master = "yarn-client",version = "1.6.0", config = list
(default = list(spark.yarn.keytab = "/home/donghua/donghua.keytab", spark.yarn.principal = "donghua@DBAGLOBE.COM")))
> sc
$master
[1] "yarn-client"

$method
[1] "shell"

$app_name
[1] "sparklyr"

$config
$config$default
$config$default$spark.yarn.keytab
[1] "/home/donghua/donghua.keytab"

$config$default$spark.yarn.principal
[1] "donghua@DBAGLOBE.COM"



$spark_home
[1] "/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/spark"

$backend
A connection with                               
description "->localhost:46015"
class       "sockconn"         
mode        "wb"               
text        "binary"           
opened      "opened"           
can read    "yes"              
can write   "yes"              

$monitor
A connection with                              
description "->localhost:8880"
class       "sockconn"        
mode        "rb"              
text        "binary"          
opened      "opened"          
can read    "yes"             
can write   "yes"             

$output_file
[1] "/tmp/RtmpXWaXfE/file7af1ca61a03_spark.log"

$spark_context
<jobj[6]>
  class org.apache.spark.SparkContext
  org.apache.spark.SparkContext@355d7d99

$java_context
<jobj[7]>
  class org.apache.spark.api.java.JavaSparkContext
  org.apache.spark.api.java.JavaSparkContext@ef616c5

attr(,"class")
[1] "spark_connection"       "spark_shell_connection" "DBIConnection"   


image

> library(DBI)
> iotdatademo <- dbGetQuery(sc, 'Select * from default.iotdatademo limit 10')
> iotdatademo

image

Reference URL: https://medium.com/@bkvarda/sparklyr-r-interface-for-spark-and-kerberos-on-cloudera-80abf5f6b4ad

60 comments:

  1. To inquire about the enormous measure of information we need an expert and all around experienced Data Scientist, to end up one of the specialists among the Data Scientist swarm, you should upgrade your range of abilities in huge information in a propelled manner. ExcelR Data Science Courses

    ReplyDelete
  2. I am genuinely thankful to the holder of this web page who has shared this wonderful paragraph at at this place
    360digitmg

    ReplyDelete
  3. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.Best data science courses in hyerabad

    ReplyDelete
  4. This is a great post. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again. photographe

    ReplyDelete
  5. I am very happy to discover your post as it will become on top in my collection of favorite blogs to visit. WordPress Developer Brisbane

    ReplyDelete
  6. Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include.
    토토사이트

    ReplyDelete
  7. I like this post,And I guess that they having fun to read this post,they shall take a good site to make a information,thanks for sharing it to me.
    파워볼사이트

    ReplyDelete
  8. Superbly written article, if only all bloggers offered the same content as you, the internet would be a far better place 스포츠토토사이트

    ReplyDelete
  9. The registered guarantor at reliable. Always fighting! I'll come often. Great work! 보증업체

    ReplyDelete
  10. I can grow up with. Please continue to do well and always do well 먹튀폴리스 I'm surprised there's such a wonderful article

    ReplyDelete
  11. I live in a different country than you This post is invaluable 카지노커뮤니티

    ReplyDelete
  12. "It’s amazing in support of me to have a site, which is good in favor of my know-how. thanks admin" 메이저놀이터

    ReplyDelete
  13. understand everything, this is really fun and enjoyable for me. Thank you guys for sharing this. Click here, 토토사이트검증

    ReplyDelete
  14. Good website! I truly love how it is easy on my eyes it is. I am wondering how I might be notified whenever a new post has been made. I have subscribed to your RSS which may do the trick? Have a great day! 먹튀사이트

    ReplyDelete
  15. Great post, you have pointed out some fantastic points , I likewise think this s a very wonderful website. i99pro

    ReplyDelete
  16. Great articles and great layout. Your blog post deserves all of the positive feedback it’s been getting. 안전놀이터

    ReplyDelete
  17. You have done a great job on this article. It’s very readable and highly intelligent. You have even managed to make it understandable and easy to read. You have some real writing talent. Thank you. cheap hosting in indiaonohosting

    ReplyDelete
  18. An fascinating discussion is value comment. I think that it is best to write extra on this matter, it won’t be a taboo topic however generally people are not enough to talk on such topics. To the next. Cheers 안전놀이터

    ReplyDelete
  19. Great post, you have pointed out some fantastic points , I likewise think this s a very wonderful website. บาคาร่า

    ReplyDelete
  20. I am thankful to you for sharing this plethora of useful information. I found this resource utmost beneficial for me. Thanks a lot for hard work. 먹튀검증

    ReplyDelete
  21. Thank you so much as you have been willing to share information with us. We will forever admire all you have done here because you have made my work as easy as ABC. best digital perm

    ReplyDelete
  22. I cannot wait to dig deep and kickoff utilizing resources that I received from you. Your exuberance is refreshing. เว็บพนันบอลดีที่สุด

    ReplyDelete
  23. Everything has its value. Thanks for sharing this informative information with us. GOOD works! here

    ReplyDelete
  24. It is my first visit to your blog, and I am very impressed with the articles that you serve. Give adequate knowledge for me. Thank you for sharing useful material. I will be back for the more great post. 안전놀이터

    ReplyDelete
  25. Good website! I truly love how it is easy on my eyes it is. I am wondering how I might be notified whenever a new post has been made. I have subscribed to your RSS which may do the trick? Have a great day! movie4me

    ReplyDelete
  26. Pretty good post. I have just stumbled upon your blog and enjoyed reading your blog posts very much. I am looking for new posts to get more precious info. Big thanks for the useful info. subcene

    ReplyDelete
  27. I really like your writing style, great information, thankyou for posting. matka tips

    ReplyDelete
  28. Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work. matka

    ReplyDelete
  29. Great post, you have pointed out some fantastic points , I likewise think this s a very wonderful website. Watergardens at Canberra

    ReplyDelete
  30. Wonderful blog! I found it while surfing around on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News? I’ve been trying for a while but I never seem to get there! Appreciate it. Pollen Collection Price

    ReplyDelete
  31. Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work. 슬롯사이트

    ReplyDelete
  32. Hi there! Nice stuff, do keep me posted when you post again something like this! bolly4u

    ReplyDelete
  33. That is the excellent mindset, nonetheless is just not help to make every sence whatsoever preaching about that mather. Virtually any method many thanks in addition to i had endeavor to promote your own article in to delicius nevertheless it is apparently a dilemma using your information sites can you please recheck the idea. thanks once more. 안전놀이터

    ReplyDelete
  34. Thank you for such a well written article. It’s full of insightful information and entertaining descriptions. Your point of view is the best among many. 먹튀검증

    ReplyDelete
  35. This is my first visit to your web journal! We are a group of volunteers and new activities in the same specialty. Website gave us helpful data to work. 무료릴게임

    ReplyDelete
  36. Thank you so much as you have been willing to share information with us. We will forever admire all you have done here because you have made my work as easy as ABC. https://biolinky.co/togelsgp

    ReplyDelete
  37. You have outdone yourself this time. It is probably the best, most short step by step guide that I have ever seen. magicien mentaliste Lyon

    ReplyDelete
  38. Wonderful blog! I found it while surfing around on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News? I’ve been trying for a while but I never seem to get there! Appreciate it. สล็อต

    ReplyDelete
  39. Great article Lot's of information to Read...Great Man Keep Posting and update to People..Thanks 메이저사이트

    ReplyDelete
  40. Wonderful blog! I found it while surfing around on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News? I’ve been trying for a while but I never seem to get there! Appreciate it. ceme deposit pulsa

    ReplyDelete
  41. Someone Sometimes with visits your blog regularly and recommended it in my experience to read as well. The way of writing is excellent and also the content is top-notch. Thanks for that insight you provide the readers! 389sports

    ReplyDelete
  42. Cool stuff you have got and you keep update all of us. sattamataka143

    ReplyDelete
  43. Nice post. I was checking constantly this blog and I’m impressed! Extremely useful info specially the last part I care for such information a lot. I was seeking this certain info for a long time. Thank you and good luck. Alliance Broadband

    ReplyDelete
  44. Join Public Mobile now with Public Mobile Promo Code E3LRK5 and get $10 off your cell phone bill using a Public Mobile Referral Code exclusively on Newsrooms. Newsrooms

    ReplyDelete
  45. Contact a Toronto lawyer for a free consultation by phone, text, WhatsApp, WeChat, Zoom or E-mail using Instant Lawyer Instant Lawyer

    ReplyDelete
  46. The post is written in very a good manner and it contains many useful information for me. buy packwoods white runtz for sale

    ReplyDelete
  47. i am always looking for some free stuffs over the internet. there are also some companies which gives free samples. order packwoods marathon Gummies

    ReplyDelete
  48. Keep up the good work , I read few posts on this web site and I conceive that your blog is very interesting and has sets of fantastic information. best merchant services company to work for

    ReplyDelete
  49. Great post, you have pointed out some fantastic points , I likewise think this s a very wonderful website. how to fix dull hair

    ReplyDelete
  50. An interesting dialogue is price comment. I feel that it is best to write more on this matter, it may not be a taboo topic however usually individuals are not enough to talk on such topics. To the next. Cheers. joker388

    ReplyDelete
  51. An interesting dialogue is price comment. I feel that it is best to write more on this matter, it may not be a taboo topic however usually individuals are not enough to talk on such topics. To the next. Cheers. sbobet asia

    ReplyDelete
  52. Cool stuff you have got and you keep update all of us. academic transcription services

    ReplyDelete
  53. liposuction is the most common cosmetic surgical procedure performed because it works, and the results are dramatic, particularly for the right patients. But are you a good candidate for liposuction? go

    ReplyDelete
  54. Most patients want to have liposuction done by a specialist. A liposuction specialist is someone who performs this procedure almost daily. get more

    ReplyDelete
  55. Impressive web site, Distinguished feedback that I can tackle. Im moving forward and may apply to my current job as a pet sitter, which is very enjoyable, but I need to additional expand. Regards. 123movie

    ReplyDelete
  56. Hi there! Nice stuff, do keep me posted when you post again something like this! 안전놀이터

    ReplyDelete
  57. Great articles and great layout. Your blog post deserves all of the positive feedback it’s been getting. บาคาร่าออนไลน์

    ReplyDelete