Get Cognitive Services Keys and Endpoint in an ARM Template

For a demo Azure Logic App I need to setup an Azure Cognitive Services account for the Computer Vision API.
I set up the resources in Azure using an ARM template so that everything is created in a reproducible way.
My logic app needs a connection to the Cognitive Services, which consists in an API Key and an endpoint.

I was not able to find a template in the Azure Quickstart Templates repo to get this information, so I find it out myself. Continue reading “Get Cognitive Services Keys and Endpoint in an ARM Template”

Mount your Azure Cloud Shell to another Storage Account

I am a heavy user of the Azure Cloud Shell, I use it every day working with the CLI 2.0 to deal with my Azure Resources.

If, like me, the first time you started the Cloud Shell and followed the wizard and clicked on the button “Create Storage”, you ended up with a new Resource Group containing a Storage Account with a random name.

I don’t like random names with my resources, so I will show you how to control this and mount the Azure Cloud Shell with a Storage Account of your choice.

Continue reading “Mount your Azure Cloud Shell to another Storage Account”

Create output variables with Azure CLI 2.0

I’m using more and more the Azure CLI 2.0, which makes my scripting life with Azure Resources a lot easier.

The default output of the Azure CLI is json, but you can also use other kind of output formats, as described in this blog.

I want to have some variables that I can reuse in other commands, so let’s say I want to get the name of all the Resource Groups which have a tag called Kind with a value VM, I can use the following JMESPath query:

The output will be:

Continue reading “Create output variables with Azure CLI 2.0”

Azure Hidden Gems: Resource Policies

Today I want to show a really useful Azure feature to help you with the governance of your Azure Subscriptions: Azure Resource Policies:

Resource policies enable you to establish conventions for resources in your organization. By defining conventions, you can control costs and more easily manage your resources. For example, you can specify that only certain types of virtual machines are allowed, or you can require that all resources have a particular tag. Policies are inherited by all child resources. So, if a policy is applied to a resource group, it is applicable to all the resources in that resource group.

How Resource Policies works

First of all, you need a policy definition. Here a description of the structure.

Let’s create one!

You can create policies definitions directly with the Azure Rest API or using PowerShell, the Azure CLI 1.0 or 2.0.

I prefer using PowerShell:

First login in Azure with: Login-AzureRmAccount and if you have more subscriptions, select the right one with Select-AzureRmSubscription.

Continue reading “Azure Hidden Gems: Resource Policies”

A better way (and script) to add a Service Principal in Azure for VSTS

From Visual Studio Team Services (VSTS) it’s possible to deploy to an Azure Subscription using an Active Directory Service Principal.

The Microsoft documentation refers to a blog post which describes a 3-clicks and a manual way to setup this principal.

For both the suggested ways (3-clicks or manual), there are some concerns from my side about the principal setup, which I think they could be improved:

  • The principal which is created during the process gets the “Contributor” role granted on the whole Azure subscription, and using the manual powershell script, the default role is even “Owner” (this can be modified).
  • The name of the Active Directory Application/Principal is some random guid which is difficult to be identified, see this picture:

Continue reading “A better way (and script) to add a Service Principal in Azure for VSTS”

Setting up VSTS with ARM Templates

In this post I want to show you what I think it’s the best way to setup VSTS working with Azure Resource Manager Templates.

Introduction

At the customer I am currently working for, we are setting up a new Azure Big Data ingestion environment and we wanted to do it using the Infrastructure as Code approach. With Azure this obviously goes with ARM Templates.

For source control, build and deployment we use Visual Studio Team Services (VSTS).

About VSTS, Build and Release Management

I have seen different setups with VSTS, some of them where the deployment take place from the build, or directly in Release Management without a build.

My approach is to have a clear separation of concerns between the Build and the Release Management.
The Build is for compiling, (Unit) Testing and creating artifacts for the deployment.
The Release Management’s responsibility is for deploying the artifacts created during the Build process. Continue reading “Setting up VSTS with ARM Templates”

VSTS Task delete Resource Group if empty

When testing deployments of Azure RM resources (ie. using the Test-AzureRmResourceGroupDeployment powershell cmdlet) an empty Resource Group is created. This Resource Group is often not needed at this point and can be deleted after you are done testing the deployment of the resources. For deleting the Resource Group you can use the VSTS task delete resource group if empty. This tasks deletes the Resource Group only if it does not contains any resources.

 

 

Dutch Azure Meetup

Dutch Azure Meetuo LogoThis week Pascal Naber, Sander Molenkamp and I started the Dutch Azure Meetup: http://www.meetup.com/Dutch-Azure-Meetup/

Azure offers a lot of services and features, and this grows almost daily. It’s hard to be up-to-date on all the Azure features. This meetup gives you the opportunity to learn about Azure in a practical way.

We want to offer a platform for Azure enthusiasts, where you can get hands-on with the technology, talk with and learn from each other. The meetup will not handle customer cases in theory but instead we talk about deep technical details, real world experiences and best practices.

Our first meetup is planned for the 2nd of June and is all about Azure Container Service:

Join the first Dutch Azure Meetup!

Docker is taking the world by storm and Microsoft Azure Container Service is where Azure and Docker meet!

During this first meetup Mark van Holsteijn will give you a brief introduction into Docker and Microsoft Container Service.After which you will have ample opportunity to get hands-on with Docker, Marathon, Azure and ASP.NET core.

We will show you how Azure Container Service provides an excellent platform for creating cross-cloud applications.

About the speaker

Mark van Holsteijn is Principal Consultant at Xebia specialized in cloud software architectures with open-source technologies.

Agenda

18:00: Dinner

18:45: Word of welcome Dutch Azure Meetup

18:50: Azure Container Service (by Mark van Holsteijn)

19:30: Hands-on with Docker and Azure Container Service

Get subscription id in an Azure Resource Manager Template

Recently I was writing an Azure Resource Manager Template to deploy a Web App. The Web App needed some Application settings (like connection strings, etc..) which I wanted to provision during the Resource Manager Template deployment. One of the settings needed was the Azure subscription id where the Web App was created.
I didn’t want to hardcode the subscription id, or provide it through a parameter (which is a way to postpone the hard-coding), so I did a little research and I found the solution.

In Azure RM Template you can use Resource Functions which can evaluate during the deployment run-time settings.

The one I needed was subscription()  which has the following structure:

This means you can use the function like this:

There are more nice functions you may need to use like: resourceGroup()resourceId ([resourceGroupName], resourceType, resourceName1, [resourceName2]…).

HDInsight error with external metastore (Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient)

Recently I was working to setup a Big Data environment in Azure.
From Azure Data Factory I was spinning up an on-demand HDInsight Cluster with an external metastore.

Unfortunately I was always getting the following error: “Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient”.
After contacting Microsoft for support about this error they found the error was caused by a known Hadoop bug:
https://issues.apache.org/jira/browse/HIVE-12536

In short the error was caused by having dashes (-) in the name of the metastore database. After removing the dashes the problem disappeared and I was able to create the on-demand HDInsight cluster.

An excerpt of the error log, the name of my metastore was db-metastore-p:


Logging initialized using configuration in file:/C:/apps/dist/hive-0.14.0.2.2.9.1-1/conf/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/apps/dist/hadoop-2.6.0.2.2.9.1-1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/apps/dist/hbase-0.98.4.2.2.9.1-1-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:445)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:619)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1483)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:63)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2743)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2762)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:426)
... 8 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1481)
... 13 more
Caused by: javax.jdo.JDOUserException: Could not create "increment"/"table" value-generation container db-metastore-p.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it.
NestedThrowables:
org.datanucleus.exceptions.NucleusUserException: Could not create "increment"/"table" value-generation container db-metastore-p.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it.