HDInsight error with external metastore (Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient)

Recently I was working to setup a Big Data environment in Azure.
From Azure Data Factory I was spinning up an on-demand HDInsight Cluster with an external metastore.

Unfortunately I was always getting the following error: “Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient”.
After contacting Microsoft for support about this error they found the error was caused by a known Hadoop bug:
https://issues.apache.org/jira/browse/HIVE-12536

In short the error was caused by having dashes (-) in the name of the metastore database. After removing the dashes the problem disappeared and I was able to create the on-demand HDInsight cluster.

An excerpt of the error log, the name of my metastore was db-metastore-p:


Logging initialized using configuration in file:/C:/apps/dist/hive-0.14.0.2.2.9.1-1/conf/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/apps/dist/hadoop-2.6.0.2.2.9.1-1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/apps/dist/hbase-0.98.4.2.2.9.1-1-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:445)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:619)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1483)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:63)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2743)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2762)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:426)
... 8 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1481)
... 13 more
Caused by: javax.jdo.JDOUserException: Could not create "increment"/"table" value-generation container db-metastore-p.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it.
NestedThrowables:
org.datanucleus.exceptions.NucleusUserException: Could not create "increment"/"table" value-generation container db-metastore-p.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it.