Sunday, November 5, 2017

Custom installation CDH5 from Parcels error with permission issue

Scenarios: Choose YARN, HDFS with optional (Spark) during the installation

Error message:

2017-11-04 23:47:56,151 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK: Permission denied
        at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:181)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:245)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK: Permission denied
        at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
        at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
        at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:944)
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:931)
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        ... 5 more
2017-11-04 23:47:56,176 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at cdh-vm.dbaglobe.com/192.168.56.10

************************************************************/

Permission incorrect for CDH folders in /var/lib

image

How to fix:

chmod 755 /var/lib/accumulo
chmod 755 /var/lib/kafka
chmod 755 /var/lib/kudu
chmod 755 /var/lib/flume-ng
chmod 755 /var/lib/hadoop-hdfs
chmod 755 /var/lib/solr
chmod 755 /var/lib/zookeeper
chmod 755 /var/lib/llama
chmod 755 /var/lib/hadoop-httpfs
chmod 755 /var/lib/hadoop-mapreduce
chmod 755 /var/lib/sqoop
chmod 755 /var/lib/hadoop-kms
chmod 755 /var/lib/hive
chmod 755 /var/lib/sqoop2
chmod 755 /var/lib/oozie
chmod 755 /var/lib/hbase
chmod 755 /var/lib/sentry
chmod 755 /var/lib/impala
chmod 755 /var/lib/spark
chmod 755 /var/lib/hadoop-yarn

chown accumulo:accumulo /var/lib/accumulo
chown kafka:kafka /var/lib/kafka
chown kudu:kudu /var/lib/kudu
chown flume:flume /var/lib/flume-ng
chown hdfs:hdfs /var/lib/hadoop-hdfs
chown solr:solr /var/lib/solr
chown zookeeper:zookeeper /var/lib/zookeeper
chown llama:llama /var/lib/llama
chown httpfs:httpfs /var/lib/hadoop-httpfs
chown mapred:mapred /var/lib/hadoop-mapreduce
chown sqoop:sqoop /var/lib/sqoop
chown kms:kms /var/lib/hadoop-kms
chown hive:hive /var/lib/hive
chown sqoop2:sqoop2 /var/lib/sqoop2
chown oozie:oozie /var/lib/oozie
chown hbase:hbase /var/lib/hbase
chown sentry:sentry /var/lib/sentry
chown impala:impala /var/lib/impala
chown spark:spark /var/lib/spark
chown yarn:yarn /var/lib/hadoop-yarn

image