java lang RuntimeException Unable to instantiate org apache hadoop hive ql metadata SessionHiveMetaStoreClient

Question

I have Hadoop 2 7 1 and apache-hive-1 2 1 versions installed on ubuntu 14 0     Why this error is occurring   Is any metastore installation required  When we typing hive command on terminal how the xml s internally called  what is the flow of those xml s  Any other configuration s required    When I am writing the hive command on ubuntu 14 0 terminal it is throwing the below exception      hive      Logging initialized using configuration in jar file  usr local hive apache-hive-1 2 1-bin lib hive-common-1 2 1 jar  hive-log4j properties     Exception in thread  main  java lang RuntimeException  java lang RuntimeException  Unable to instantiate org apache hadoop hive ql metadata SessionHiveMetaStoreClient         at org apache hadoop hive ql session SessionState start SessionState java 522          at org apache hadoop hive cli CliDriver run CliDriver java 677          at org apache hadoop hive cli CliDriver main CliDriver java 621          at sun reflect NativeMethodAccessorImpl invoke0 Native Method          at sun reflect NativeMethodAccessorImpl invoke NativeMethodAccessorImpl java 62          at sun reflect DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl java 43          at java lang reflect Method invoke Method java 520          at org apache hadoop util RunJar run RunJar java 221          at org apache hadoop util RunJar main RunJar java 136      Caused by  java lang RuntimeException  Unable to instantiate org apache hadoop hive ql metadata SessionHiveMetaStoreClient         at org apache hadoop hive metastore MetaStoreUtils newInstance MetaStoreUtils java 1523          at org apache hadoop hive metastore RetryingMetaStoreClient  lt init gt  RetryingMetaStoreClient java 86          at org apache hadoop hive metastore RetryingMetaStoreClient getProxy RetryingMetaStoreClient java 132          at org apache hadoop hive metastore RetryingMetaStoreClient getProxy RetryingMetaStoreClient java 104          at org apache hadoop hive ql metadata Hive createMetaStoreClient Hive java 3005          at org apache hadoop hive ql metadata Hive getMSC Hive java 3024          at org apache hadoop hive ql session SessionState start SessionState java 503              8 more     Caused by  java lang reflect InvocationTargetException         at sun reflect NativeConstructorAccessorImpl newInstance0 Native Method          at sun reflect NativeConstructorAccessorImpl newInstance NativeConstructorAccessorImpl java 62          at sun reflect DelegatingConstructorAccessorImpl newInstance DelegatingConstructorAccessorImpl java 45          at java lang reflect Constructor newInstance Constructor java 426          at org apache hadoop hive metastore MetaStoreUtils newInstance MetaStoreUtils java 1521              14 more     Caused by  javax jdo JDOFatalInternalException  Error creating transactional connection factory     NestedThrowables      java lang reflect InvocationTargetException         at org datanucleus api jdo NucleusJDOHelper getJDOExceptionForNucleusException NucleusJDOHelper java 587          at org datanucleus api jdo JDOPersistenceManagerFactory freezeConfiguration JDOPersistenceManagerFactory java 788          at org datanucleus api jdo JDOPersistenceManagerFactory createPersistenceManagerFactory JDOPersistenceManagerFactory java 333          at org datanucleus api jdo JDOPersistenceManagerFactory getPersistenceManagerFactory JDOPersistenceManagerFactory java 202          at sun reflect NativeMethodAccessorImpl invoke0 Native Method          at sun reflect NativeMethodAccessorImpl invoke NativeMethodAccessorImpl java 62          at sun reflect DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl java 43          at java lang reflect Method invoke Method java 520          at javax jdo JDOHelper 16 run JDOHelper java 1965          at java security AccessController doPrivileged Native Method          at javax jdo JDOHelper invoke JDOHelper java 1960          at javax jdo JDOHelper invokeGetPersistenceManagerFactoryOnImplementation JDOHelper java 1166          at javax jdo JDOHelper getPersistenceManagerFactory JDOHelper java 808          at javax jdo JDOHelper getPersistenceManagerFactory JDOHelper java 701          at org apache hadoop hive metastore ObjectStore getPMF ObjectStore java 365          at org apache hadoop hive metastore ObjectStore getPersistenceManager ObjectStore java 394          at org apache hadoop hive metastore ObjectStore initialize ObjectStore java 291          at org apache hadoop hive metastore ObjectStore setConf ObjectStore java 258          at org apache hadoop util ReflectionUtils setConf ReflectionUtils java 76          at org apache hadoop util ReflectionUtils newInstance ReflectionUtils java 136          at org apache hadoop hive metastore RawStoreProxy  lt init gt  RawStoreProxy java 57          at org apache hadoop hive metastore RawStoreProxy getProxy RawStoreProxy java 66          at org apache hadoop hive metastore HiveMetaStore HMSHandler newRawStore HiveMetaStore java 593          at org apache hadoop hive metastore HiveMetaStore HMSHandler getMS HiveMetaStore java 571          at org apache hadoop hive metastore HiveMetaStore HMSHandler createDefaultDB HiveMetaStore java 624          at org apache hadoop hive metastore HiveMetaStore HMSHandler init HiveMetaStore java 461          at org apache hadoop hive metastore RetryingHMSHandler  lt init gt  RetryingHMSHandler java 66          at org apache hadoop hive metastore RetryingHMSHandler getProxy RetryingHMSHandler java 72          at org apache hadoop hive metastore HiveMetaStore newRetryingHMSHandler HiveMetaStore java 5762          at org apache hadoop hive metastore HiveMetaStoreClient  lt init gt  HiveMetaStoreClient java 199          at org apache hadoop hive ql metadata SessionHiveMetaStoreClient  lt init gt  SessionHiveMetaStoreClient java 74              19 more     Caused by  java lang reflect InvocationTargetException         at sun reflect NativeConstructorAccessorImpl newInstance0 Native Method          at sun reflect NativeConstructorAccessorImpl newInstance NativeConstructorAccessorImpl java 62          at sun reflect DelegatingConstructorAccessorImpl newInstance DelegatingConstructorAccessorImpl java 45          at java lang reflect Constructor newInstance Constructor java 426          at org datanucleus plugin NonManagedPluginRegistry createExecutableExtension NonManagedPluginRegistry java 631          at org datanucleus plugin PluginManager createExecutableExtension PluginManager java 325          at org datanucleus store AbstractStoreManager registerConnectionFactory AbstractStoreManager java 282          at org datanucleus store AbstractStoreManager  lt init gt  AbstractStoreManager java 240          at org datanucleus store rdbms RDBMSStoreManager  lt init gt  RDBMSStoreManager java 286          at sun reflect NativeConstructorAccessorImpl newInstance0 Native Method          at sun reflect NativeConstructorAccessorImpl newInstance NativeConstructorAccessorImpl java 62          at sun reflect DelegatingConstructorAccessorImpl newInstance DelegatingConstructorAccessorImpl java 45          at java lang reflect Constructor newInstance Constructor java 426          at org datanucleus plugin NonManagedPluginRegistry createExecutableExtension NonManagedPluginRegistry java 631          at org datanucleus plugin PluginManager createExecutableExtension PluginManager java 301          at org datanucleus NucleusContext createStoreManagerForProperties NucleusContext java 1187          at org datanucleus NucleusContext initialise NucleusContext java 356          at org datanucleus api jdo JDOPersistenceManagerFactory freezeConfiguration JDOPersistenceManagerFactory java 775              48 more     Caused by  org datanucleus exceptions NucleusException  Attempt to invoke the  BONECP  plugin to create a ConnectionPool gave an error   The specified datastore driver   com mysql jdbc Driver   was not found in the CLASSPATH  Please check your CLASSPATH specification  and the name of the driver          at org datanucleus store rdbms ConnectionFactoryImpl generateDataSources ConnectionFactoryImpl java 259          at org datanucleus store rdbms ConnectionFactoryImpl initialiseDataSources ConnectionFactoryImpl java 131          at org datanucleus store rdbms ConnectionFactoryImpl  lt init gt  ConnectionFactoryImpl java 85              66 more     Caused by  org datanucleus store rdbms connectionpool DatastoreDriverNotFoundException  The specified datastore driver   com mysql jdbc Driver   was not found in the CLASSPATH  Please check your CLASSPATH specification  and the name of the driver          at org datanucleus store rdbms connectionpool AbstractConnectionPoolFactory loadDriver AbstractConnectionPoolFactory java 58          at org datanucleus store rdbms connectionpool BoneCPConnectionPoolFactory createConnectionPool BoneCPConnectionPoolFactory java 54          at org datanucleus store rdbms ConnectionFactoryImpl generateDataSources ConnectionFactoryImpl java 238              68 more   To avoid above error I created hive-site xml with      lt configuration gt          lt property gt           lt name gt hive metastore warehouse dir lt  name gt           lt value gt  home local hive-metastore-dir warehouse lt  value gt          lt  property gt        lt property gt         lt name gt javax jdo option ConnectionURL lt  name gt         lt value gt jdbc mysql   localhost 3306 hivedb createDatabaseIfNotExist true lt  value gt       lt  property gt       lt property gt         lt name gt javax jdo option ConnectionDriverName lt  name gt         lt value gt com mysql jdbc Driver lt  value gt       lt  property gt       lt property gt         lt name gt javax jdo option ConnectionUserName lt  name gt         lt value gt user lt  value gt       lt  property gt       lt property gt         lt name gt javax jdo option ConnectionPassword lt  name gt         lt value gt password lt  value gt       lt  property gt        lt configuration gt    Also provided the environment variables in    bashrc file  Still the error persist   HIVE home directory configuration export HIVE HOME  usr local hive apache-hive-1 2 1-bin export PATH   PATH  HIVE HOME bin

User · Answer

starting the hive metastore service worked for me. First, set up the database for hive metastore:

 $ hive --service metastore

` https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/validate_installation.html

Second, run the following commands:

 $ schematool -dbType mysql -initSchema  
 $ schematool -dbType mysql -info

https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool

User · Answer

Run hive in debug mode  hive -hiveconf hive root logger DEBUG console  and then execute   show tables  can find the actual problem

User · Answer

I solved this problem by removing --deploy-mode cluster from spark-submit code   By default   spark submit takes client mode which has following advantage     1  It opens up Netty HTTP server and distributes all jars to the worker nodes  2  Driver program runs on master node   which means dedicated resources to driver process    While in cluster mode      1   It runs on worker node   2  All the jars need to be placed in a common folder of the cluster so that it is accessible to all the worker nodes or in folder of each worker node    Here it s not able to access hive metastore due to unavailability of hive jar to any of the nodes in cluster

User · Answer

I did below modifications and I am able to start the Hive Shell without any errors  1     bashrc Inside bashrc file add the below environment variables at End Of File   sudo gedit    bashrc  Java Home directory configuration export JAVA HOME  quot  usr lib jvm java-9-oracle quot  export PATH  quot  PATH  JAVA HOME bin quot     Hadoop home directory configuration export HADOOP HOME  usr local hadoop export PATH  PATH  HADOOP HOME bin export PATH  PATH  HADOOP HOME sbin  export HIVE HOME  usr lib hive export PATH  PATH  HIVE HOME bin  2  hive-site xml You have to create this file hive-site xml  in conf directory of Hive and add the below details  lt  xml version  quot 1 0 quot  encoding  quot UTF-8 quot  standalone  quot no quot   gt   lt  xml-stylesheet type  quot text xsl quot  href  quot configuration xsl quot   gt   lt configuration gt    lt property gt     lt name gt javax jdo option ConnectionURL lt  name gt     lt value gt jdbc mysql   localhost metastore createDatabaseIfNotExist true lt  value gt   lt  property gt     lt property gt     lt name gt javax jdo option ConnectionDriverName lt  name gt     lt value gt com mysql jdbc Driver lt  value gt   lt  property gt    lt property gt     lt name gt javax jdo option ConnectionUserName lt  name gt     lt value gt root lt  value gt   lt  property gt    lt property gt     lt name gt javax jdo option ConnectionPassword lt  name gt     lt value gt root lt  value gt   lt  property gt    lt property gt     lt name gt datanucleus autoCreateSchema lt  name gt     lt value gt true lt  value gt   lt  property gt    lt property gt     lt name gt datanucleus fixedDatastore lt  name gt     lt value gt true lt  value gt   lt  property gt    lt property gt    lt name gt datanucleus autoCreateTables lt  name gt    lt value gt True lt  value gt    lt  property gt    lt  configuration gt   3  You also need to put the jar file mysql-connector-java-5 1 28 jar  in the lib directory of Hive 4  Below installations required on your Ubuntu to Start the Hive Shell   MySql Hadoop Hive Java  5  Execution Part   Start all services of Hadoop  start-all sh  Enter the jps command to check whether all Hadoop services are up and running  jps  Enter the hive command to enter into hive shell  hive

User · Answer

This is probably due to its lack of connections to the Hive Meta Store my hive Meta Store is stored in Mysql so I need to visit Mysql So I add a dependency in my build sbt libraryDependencies     quot mysql quot     quot mysql-connector-java quot     quot 5 1 38 quot   and the problem is solved

User · Answer

I have used MySQL DB for Hive MetaStore  Please follow the below steps     in hive-site xml the metastore should be proper    lt property gt    lt name gt javax jdo option ConnectionURL lt  name gt     lt value gt jdbc mysql   localhost metastorecreateDatabaseIfNotExist true amp amp useSSL false lt  value gt   lt  property gt     go to the mysql  mysql -u hduser -p then run drop database metastore then come out from MySQL and execute schematool -initSchema dbType mysql   Now error will go

User · Answer

mybe your hive metastore are inconsistent  I m in this scene  first  I run    schematool -dbType mysql -initSchema    then I found this   Error  Duplicate key name  PCS STATS IDX   state 42000 code 1061  org apache hadoop hive metastore HiveMetaException  Schema initialization FAILED  Metastore state would be inconsistent    then I run    schematool -dbType mysql -info  found this error  Hive distribution version      2 3 0 Metastore schema version     1 2 0 org apache hadoop hive metastore HiveMetaException  Metastore schema version is not compatible  Hive Version  2 3 0  Database Schema Version  1 2 0   so i format my hive metastore  then it s done   drop mysql database  the database named hive db run schematool -dbType mysql -initSchema for initialize metadata

User · Answer

In my case when i tried    hive --service metastore    I got      MetaException message Version information not found in metastore      The necessary tables required for the metastore are missing in MySQL  Manually create the tables and restart hive metastore   cd  HIVE HOME scripts metastore upgrade mysql     lt  Login into MySQL  gt    mysql gt  drop database IF EXISTS  lt metastore db name gt    mysql gt  create database  lt metastore db name gt    mysql gt  use  lt metastore db name gt    mysql gt  source hive-schema-2 x x mysql sql     metastore db name should match the database name mentioned in hive-site xml files connection property tag   hive-schema-2 x x mysql sql file depends on the version available in the current directory  Try to go for the latest because it holds many old schema files also   Now try to execute hive --service metastore  If everything goes cool  then simply start the hive from terminal    gt hive   I hope the above answer serves your need

User · Answer

I have also faced this problem but i had restart Hadoop and use command hadoop dfsadmin -safemode leave  now start hive it will work i think

User · Answer

In my case  I stopped my docker hive container and run it again and finally  it worked  Hope it will be useful for someone  Note  This might be caused because there might be an instance running in the background so stopping the container will stop all background instances

User · Answer

just open the hive terminal from the hive folder after editing  bashrc  and  hive-site xml  files  Steps-- open hive folder where it is installed  now open terminal from folder

User · Answer

In the middle of the stack trace  lost in the  reflection  junk  you can find the root cause   The specified datastore driver   com mysql jdbc Driver   was not found in the CLASSPATH  Please check your CLASSPATH specification  and the name of the driver

User · Answer

If you re just playing around in local mode  you can drop metastore DB and reinstate it   rm -rf metastore db   HIVE HOME bin schematool -initSchema -dbType derby

[apache] java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Examples related to apache

Examples related to hadoop

Examples related to hive