Auto-scale GitHub Runners (and ADO Agents) with Azure Container Apps Jobs and Workload Profiles

Intro

In this blog I want to explain how to run your private GitHub Runners (this should work for ADO Agents as well) using the new Azure Container Apps Jobs (in preview) functionality with Workload Profiles (also in preview) using auto-scaling with KEDA.
With this setup there’s no need for a complicated container (AKS) infrastructure deployment (and maintenance!) with enough flexibility, scalability and cost saving.

Container Apps and Jobs don’t support running Docker in containers. Any steps in your workflows that use Docker commands will fail when run on a self-hosted runner or agent in a Container Apps Job.

I have created a public GitHub repo where you can find all the code related to this post, mostly in Terraform (I have added also Azure CLI scripts).

While I was writing this post, Microsoft published a post like this one, but in the examples there’s no Workload Profile functionality with VNET Integration which supports user defined routes (UDR) and egress through NAT Gateway (or Azure Firewall in a hub and spoke setup) which make possible to run the runners in a private environment, to connect to your Azure resources without public access.

Using Azure Container App Jobs is the perfect use-case for this kind of scenarios, the Jobs are made for on-demand processing, exactly what GitHub Runners jobs are.

Continue reading “Auto-scale GitHub Runners (and ADO Agents) with Azure Container Apps Jobs and Workload Profiles” →
0

Using Azure Managed Identity with .Net Core on Azure DevOps Agents

During my last project I needed to run some integration test written in .Net Core 2.2 in an Azure Devops Pipeline.

The code needed some secrets from an Azure KeyVault and doing some other stuff on other Azure Resources using Azure Managed Identities for authentication on them.

In .Net Core you can easily accomplish this using the AppAuthentication Nuget library.

Here an example how to use this library for getting secrets etc.. from the KeyVault.

To run this code you need an Azure infrastructure where Managed Identities is enabled, like a VM, Azure Web App/Function App etc….

So, how can you run this code on Azure DevOps agents?

Continue reading “Using Azure Managed Identity with .Net Core on Azure DevOps Agents” →
0

Using Git submodules in (private) Azure DevOps repositories

I often use Git Submodules in my repositories, it’s a handy way to have external source code in your repo, without having to worry much about the external updates and source management (you can just checkout, pull etc as every normal git repo).

Often the external sources are hosted in public GitHub repos, without any need to authenticate.
You create .gitmodules file look like:†

[submodule "DbConnector"]
	path = DbConnector
	url = https://github.com/chaconinc/DbConnector

And with a few more command you are ready to go.

Lastly I wanted to use submodules between my private Azure DevOps repositories in the same Azure DevOps Project.
So I added the submodule with its full path, like:

Continue reading “Using Git submodules in (private) Azure DevOps repositories” →
0

Solve “System.Threading.Tasks.TaskCanceledException : The operation was canceled. ” on Ubuntu 18.04 in .Net Core 2.2

I am running some .Net Core 2.2 (at the moment of writing version 2.2.301) integration test in Azure DevOps (ADO) running on my private Linux Ubuntu 18.04 agents.
The tests are quiet simple, just call an internally exposed Rest API through Azure API Management.
Because the APIM is running internally the endpoint are using self-signed SSL certificates generated with an internal Root Certificate.

During the runs in ADO I was experiencing the following error: 

Error Message:
   System.Threading.Tasks.TaskCanceledException : The operation was canceled.
  Stack Trace:
     at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
Continue reading “Solve “System.Threading.Tasks.TaskCanceledException : The operation was canceled. ” on Ubuntu 18.04 in .Net Core 2.2″ →
0

Run Elasticsearch image on Azure Container Instances

Last week I was moving a Docker application to Azure Container Instances. The application consisted out of 3 containers and one of them was using the official image of Elasticsearch: 

FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.0

Unlucky the container was failing to start, giving the following error: 

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

After a quick search, I found this post where someone else was having the same problem.

Continue reading “Run Elasticsearch image on Azure Container Instances” →
1

Make your VSTS extensions smarter with Endpoints Datasource bindings

At the current customer I am working for I am creating a lot of VSTS Extensions to deploy Azure Resources.
I want my tasks to be so much user friendly as possible, and one of the things to accomplish this is with Data Source Bindings.

What are Data Source Bindings?

Data source bindings bind a drop-down input field in the UI (e.g. task input) which needs to be dynamically populated with values from ta REST API that needs to be invoked to fetch the list of values.

Where are the Data Sources defined?

The Data sources are defined in the Service Endpoints.

Service endpoints support querying data from external services through REST API.

The data queried can be used to populate task input drop-downs. Continue reading “Make your VSTS extensions smarter with Endpoints Datasource bindings” →

1

Create a sharded MongoDB in Azure Cosmos DB

During my last project I was setting up a Release Pipeline in VSTS and one of the steps was to create a Sharded MongoDB in Cosmos DB.

I am a big fan of the Azure CLI, which I use quiet often.

I started to create in bash an Azure CLI script but soon I discovered that it was not working well. The script gave no errors, everything seemed to be ok, but I was getting an error when inserting a document that the sharded key was not found/provided. Continue reading “Create a sharded MongoDB in Azure Cosmos DB” →

1

Get Cognitive Services Keys and Endpoint in an ARM Template

For a demo Azure Logic App I need to setup an Azure Cognitive Services account for the Computer Vision API.
I set up the resources in Azure using an ARM template so that everything is created in a reproducible way.
My logic app needs a connection to the Cognitive Services, which consists in an API Key and an endpoint.

I was not able to find a template in the Azure Quickstart Templates repo to get this information, so I find it out myself. Continue reading “Get Cognitive Services Keys and Endpoint in an ARM Template” →

0

Mount your Azure Cloud Shell to another Storage Account

I am a heavy user of the Azure Cloud Shell, I use it every day working with the CLI 2.0 to deal with my Azure Resources.

If, like me, the first time you started the Cloud Shell and followed the wizard and clicked on the button “Create Storage”, you ended up with a new Resource Group containing a Storage Account with a random name.

I don’t like random names with my resources, so I will show you how to control this and mount the Azure Cloud Shell with a Storage Account of your choice.

Continue reading “Mount your Azure Cloud Shell to another Storage Account” →

1

Blog at WordPress.com.

Up ↑