Thursday, December 29, 2016

Access HBase from BDA Lite using Oracle BigData SQL

SQL> select * from hbase_datademo where rowkey=1000001451580500;
select * from hbase_datademo where rowkey=1000001451580500
*
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-11504: error from external driver: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes


[oracle@bigdatalite bigdatasql_config]$ less /tmp/bigdata-log4j.log
16/12/29 11:25:41 INFO xcat.XCat: cluster property com.oracle.bigdata.conf.default.serdepropcacheenabled is true
16/12/29 11:25:41 DEBUG xcat.XCat: access param com.oracle.bigdata.serdepropcache is true
16/12/29 11:25:41 DEBUG hive.XCatHive: XCatHive.open: connecting to the hive metastore...
16/12/29 11:25:41 DEBUG hive.XCatHive: XCatHive.open: connect succeeded
16/12/29 11:25:41 DEBUG hive.XCatHive: XCatHive.open: getTable succeeded
16/12/29 11:25:41 ERROR sql.JXADDescribe: JXADDescribe.getInfo
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
        at org.apache.hadoop.hive.hbase.HBaseSerDe.parseColumnsMapping(HBaseSerDe.java:184)
        at org.apache.hadoop.hive.hbase.HBaseSerDeParameters.(HBaseSerDeParameters.java:73)
        at org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:117)
        at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:53)
        at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:520)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:392)
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:273)
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:255)
        at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:602)
        at oracle.hadoop.sql.xcat.hive.XCatHiveUtil.extractSchema(XCatHiveUtil.java:205)
        at oracle.hadoop.sql.xcat.hive.XCatHiveUtil.valueOf(XCatHiveUtil.java:276)
        at oracle.hadoop.sql.xcat.hive.XCatHive.open(XCatHive.java:209)
        at oracle.hadoop.sql.JXADDescribe._compile(JXADDescribe.java:491)
        at oracle.hadoop.sql.JXADDescribe.getXaddoc(JXADDescribe.java:621)
        at oracle.hadoop.sql.JXADDescribe._getInfo(JXADDescribe.java:429)
        at oracle.hadoop.sql.JXADDescribe.getInfo(JXADDescribe.java:380)
16/12/29 11:25:41 DEBUG metrics.TaskMetrics: jxad_task: DESCRIBE__GET_XADDOC_REQTYPE, time: 64, bytes: 0
16/12/29 11:25:41 TRACE sql.JXADProvider: Entering kubsagtiJNIConvCB...
16/12/29 11:25:41 TRACE sql.JXADProvider: Leaving kubsagtiJNIConvCB...retSize=0
16/12/29 11:25:41 TRACE sql.JXADProvider: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
16/12/29 11:25:41 ERROR sql.JXADProvider: KUP-11504: error from external driver: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
16/12/29 11:25:41 TRACE sql.JXADProvider: Leaving kubsjniDescribeGetInfo...status=6
16/12/29 11:25:41 TRACE sql.JXADProvider: Leaving kubsagtiDescribeGetInfo...status=33
16/12/29 11:25:41 TRACE sql.JXADProvider: Leaving kubsagtDescribeGetInfo...status=33
16/12/29 11:25:41 DEBUG sql.JXADProvider: using agent handle=79
16/12/29 11:25:41 DEBUG sql.JXADProvider: using agent context=0xedfe98
16/12/29 11:25:41 DEBUG sql.JXADProvider: using agent jvm pointer=0x7fe020bf1b48
16/12/29 11:25:41 DEBUG sql.JXADProvider: using agent jni handle=0x10009f8

SQL> select * from hbase_datademo where rownum<10 p="">select * from hbase_datademo where rownum<10 p="">              *
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-11504: error from external driver:
oracle.hadoop.sql.xcat.common.XCatException : 1001 : Error getting splits
document. Cause : java.io.IOException:
java.lang.reflect.InvocationTargetException

16/12/29 13:40:15 DEBUG hadoop.XCatInfm: listStatus(final) file =hdfs://bigdatalite.localdomain:8020/user/hive/warehouse/hbase_datademo
16/12/29 13:40:15 ERROR sql.JXADDescribe: JXADDescribe.getInfo
oracle.hadoop.sql.xcat.common.XCatException : 1001 : Error getting splits document. Cause : java.io.IOException: java.lang.reflect.InvocationTargetException
        at oracle.hadoop.sql.xcat.XCatInfo.getSplitsDoc(XCatInfo.java:395)
        at oracle.hadoop.sql.JXADDescribe.getSplitsDoc(JXADDescribe.java:545)
        at oracle.hadoop.sql.JXADDescribe._getInfo(JXADDescribe.java:417)
        at oracle.hadoop.sql.JXADDescribe.getInfo(JXADDescribe.java:380)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
        at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplitsInternal(HiveHBaseTableInputFormat.java:463)
        at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits(HiveHBaseTableInputFormat.java:450)
        at oracle.hadoop.sql.xcat.hadoop.mapred.XCatMapredInfm.computeBaseSplits(XCatMapredInfm.java:169)
        at oracle.hadoop.sql.xcat.hadoop.XCatGetSplits.computeSplits(XCatGetSplits.java:563)
        at oracle.hadoop.sql.xcat.XCatInfo.getSplits(XCatInfo.java:354)
        at oracle.hadoop.sql.xcat.XCatInfo.getSplitsDoc(XCatInfo.java:384)
        ... 3 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
        ... 11 more
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:217)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:419)
        at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:919)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:657)
        ... 16 more
16/12/29 13:40:15 DEBUG metrics.TaskMetrics: jxad_task: DESCRIBE__GET_SPLITS_DOC_REQTYPE, time: 59, bytes: 0
16/12/29 13:40:15 TRACE sql.JXADProvider: Entering kubsagtiJNIConvCB...

How to Fix: Adding Hbase and Hbase Lib to java class path for hadoop inside bigdata.properties file

[oracle@bigdatalite tmp]$ diff /u01/bigdatasql_config/bigdata.properties /u01/bigdatasql_config/bigdata.properties.orig
3c3
< java.classpath.hadoop=/usr/lib/hadoop/client/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hbase/*:/usr/lib/hbase/lib/*
---
> java.classpath.hadoop=/usr/lib/hadoop/client/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-mapreduce/lib/*

--- restart BDA Lite