Showing posts with label Atlas. Show all posts
Showing posts with label Atlas. Show all posts

Thursday, February 28, 2019

How to import Hive metadata to Atlas

How to import existing hive tables which created before Apache Atlas added?

[hive@hdp ~]$ /usr/hdp/current/atlas-server/hook-bin/import-hive.sh
Using Hive configuration directory [/etc/hive/conf]
Log file for import is /usr/hdp/current/atlas-server/logs/import-hive.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2019-02-28T18:20:05,890 INFO [main] org.apache.atlas.ApplicationProperties - Looking for atlas-application.properties in classpath
2019-02-28T18:20:05,897 INFO [main] org.apache.atlas.ApplicationProperties - Loading atlas-application.properties from file:/etc/hive/3.1.0.0-78/0/atlas-application.properties
2019-02-28T18:20:06,006 INFO [main] org.apache.atlas.ApplicationProperties - No graphdb backend specified. Will use 'janus'
2019-02-28T18:20:06,006 INFO [main] org.apache.atlas.ApplicationProperties - Using storage backend 'hbase2'
2019-02-28T18:20:06,006 INFO [main] org.apache.atlas.ApplicationProperties - Using index backend 'solr'
2019-02-28T18:20:06,007 INFO [main] org.apache.atlas.ApplicationProperties - Setting solr-wait-searcher property 'true'
2019-02-28T18:20:06,007 INFO [main] org.apache.atlas.ApplicationProperties - Setting index.search.map-name property 'false'
2019-02-28T18:20:06,011 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.db-cache = true
2019-02-28T18:20:06,011 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.db-cache-clean-wait = 20
2019-02-28T18:20:06,012 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.db-cache-size = 0.5
2019-02-28T18:20:06,012 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.tx-cache-size = 15000
2019-02-28T18:20:06,012 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.tx-dirty-size = 120
Enter username for atlas :- admin
Enter password for atlas :- 
2019-02-28T18:20:10,984 INFO [main] org.apache.atlas.AtlasBaseClient - Client has only one service URL, will use that for all actions: http://hdp.dbaglobe.com:21000
2019-02-28T18:20:11,028 INFO [main] org.apache.hadoop.hive.conf.HiveConf - Found configuration file file:/etc/hive/3.1.0.0-78/0/hive-site.xml
2019-02-28T18:20:12,131 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.fetch.partition.stats does not exist
2019-02-28T18:20:12,131 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.heapsize does not exist
2019-02-28T18:20:13,617 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-02-28T18:20:14,051 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Trying to connect to metastore with URI thrift://hdp:9083
2019-02-28T18:20:14,223 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Opened a connection to metastore, current connections: 1
2019-02-28T18:20:14,474 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Connected to metastore.
2019-02-28T18:20:14,474 INFO [main] org.apache.hadoop.hive.metastore.RetryingMetaStoreClient - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive (auth:SIMPLE) retries=24 delay=5 lifetime=0
2019-02-28T18:20:15,314 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing Hive metadata
2019-02-28T18:20:15,356 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 4 databases
2019-02-28T18:20:15,717 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:16,494 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Database default is already registered - id=620ebb57-a216-469f-8184-3def1b22da16. Updating it.
2019-02-28T18:20:16,711 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:16,815 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Updated hive_db entity: name=default@lake, guid=620ebb57-a216-469f-8184-3def1b22da16
2019-02-28T18:20:16,848 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - No tables to import in database default
2019-02-28T18:20:16,894 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:16,896 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Database demodb is already registered - id=2ebf0dce-e258-4121-ac3f-16e38facce2e. Updating it.
2019-02-28T18:20:16,923 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:16,923 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Updated hive_db entity: name=demodb@lake, guid=2ebf0dce-e258-4121-ac3f-16e38facce2e
2019-02-28T18:20:16,928 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 5 tables to import in database demodb
2019-02-28T18:20:17,184 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=404
2019-02-28T18:20:17,234 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.fetch.partition.stats does not exist
2019-02-28T18:20:17,234 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.heapsize does not exist
2019-02-28T18:20:23,463 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:23,604 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/guid/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:23,607 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Created hive_table entity: name=demodb.person@lake, guid=af210ef7-9c1c-4faf-b07e-363dc51683aa
2019-02-28T18:20:23,637 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=404
2019-02-28T18:20:25,496 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:25,573 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/guid/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:25,575 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Created hive_table entity: name=demodb.zip@lake, guid=8aad59c6-ed05-43c9-8d89-e74ac712ee2b
2019-02-28T18:20:25,670 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:20:25,673 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Table demodb.position is already registered with id 5476f1f0-8175-4b50-873b-e6589d0313bf. Updating entity.
2019-02-28T18:20:26,268 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
 ...

2019-02-28T18:21:45,219 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Updated hive_table entity: name=sys.wm_mappings@lake, guid=505dd84a-0221-433d-bc9e-5a406062a48c
2019-02-28T18:21:45,232 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2019-02-28T18:21:45,232 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Process sys.wm_mappings@lake:1549277088000 is already registered
2019-02-28T18:21:45,232 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Successfully imported 45 tables from database sys
Hive Meta Data imported successfully!!!