If you use Azure services for your mission critical apps, you will want to know when Azure is having an outage that affects you. The Azure health status page is pretty much useless since it usually gets updated hours after an outage occurs. The outage has to affect a large enough percentage of users before it gets reported on the main health status page. There is nothing more frustrating when you know for a fact that there is an outage affecting your apps, only to see the health status page report that "all services are operating normally". This is where Azure Monitor comes in. Azure Monitor promises to alert you sooner and only for outages that affect your subscription which is pretty helpful since I don't care about outages in services/datacenters that I don't use.

Setup

In the Azure portal, go to the Monitor blade, select Alerts, then Add activity log alert. You should see something like this:

For alert criteria, make sure to choose Service Health. In this case, I'm interested in all alert types and levels under the Service Health category.

Under alert actions, the most straightforward options are SMS and email. But in this case, we want to forward these alerts to a Microsoft Teams channel for greater visibility within the organization. This is possible by using the Webhook option. To do this, we can provision a Function or Logic app to consume the Webhook payload, then forward that to a Teams channel of our choosing (or even Slack). That's too much work though. Thankfully, Microsoft Teams lets you send a message to a specific channel via email. We'll use this option instead since its a lot easier. Just go to the channel you want the alerts to go to, click on the three dots next to the channel's name and select "Get email address".

Simply copy and paste this email in the alert form using the email option. That's it! With very little setup, you will now get alerted in Teams for outages that affect your services. This will hopefully help you respond to outages in a more timely manner.

Further resources

• Azure Monitor documentation
• Azure Monitor Notifications (Channel 9 video)

Application Insights is an Azure offering that helps developers gain insight into their applications as far as exceptions, page views, response times..etc. However, one less known use case of Application Insights is the ability to track domain related events.

My team and I at Shift Digital work on Enterprise Lead Management Systems for automotive OEMs and other clients. Our systems accept leads from a variety of sources and delivers them to dealerships. We recently started using Application Insights to keep a close eye on what's happening within our systems. This helped us get better visibility on exceptions, response times, etc.. but we soon discovered how we can use the custom events feature of Application Insights to track domain-related telemetry.

For example, we wanted to know the number of leads we are getting in real time. How many leads are accepted? How many are rejected? We never had that near real time insight into our application before. So with a few lines of code.. we were able to generate graphs like these:

The code

To make this possible, we simply track a custom event with properties and metrics as shown below:

Note: we first need to install the Microsoft.ApplicationInsights Nuget package

PM> Install-Package Microsoft.ApplicationInsights  

//capture received time
var receivedTime = DateTime.Now();

//initialize telemetry client
var telemetryClient = new TelemetryClient();  
telemetryClient.Context.InstrumentationKey = "your telemetry key";  
telemetryClient.Context.Operation.Name = "LeadProcessing";

//do the actual work
...

//calculate how long it took to do the work
var processingTime = DateTime.Now.Subtract(receivedTime).TotalSeconds;

//track the event
var properties = new Dictionary<string, string>  
{
   { "Source", "the source of the lead" }
};

var metrics = new Dictionary<string, double>  
{ 
  { "LeadProcessingTime", processingTime } 
};

telemetryClient.TrackEvent("LeadProcessed", properties, metrics);  

The event being tracked here is "LeadProcessed". In addition to the event itself, we can also pass properties and metrics. Properties help when filtering and searching for events. For example, we are tracking the source of the lead so we can break leads by source when reporting on that event. On the other hand, metrics let us measure aspects related to the event. In our case, we are measuring how long it took us to process the lead. Both properties and metrics are optional.

Reporting on events

The easiest way I've found to generate these graphs is to pin an empty graph to an Azure dashboard, then customizing it. I turn grouping on by event name, then filter out any events I don't want.

Another way to report on these events is by using the Analytics feature of Application Insights where you can write custom queries to drill into the exact events we want to report on. For example, if we want a pie chart to visualize how many leads were processed versus rejected in the last day, we can write the following query:

customEvents  
| where name == "LeadProcessed"  or  name == "LeadRejected" and timestamp <= ago(1d) 
| render piechart   

which gives us the following graph:

The Analytics tool uses a special query syntax that combines pipes, lots of helper methods, and intellisense to make querying data easier. These graphs can easily be pinned to an Azure dashboard which is nice.

Alerts for custom events with metrics

We can even set up alerts for metrics in our custom events. For example, if we want to be alerted when LeadProcessingTime exceeds a certain threshold, all we have to do is add an alert rule to the Application Insights resource like this:

Now every time this metric exceeds our threshold for 15 minutes, we get an email alert. We can also specify a Webhook to be invoked if we want to forward this alert to Slack or HipChat for example. Neat!

Conclusion

Application Insights is a powerful tool that can help us get real-time insight into domain events happening within our application. We saw above how we were able to generate useful graphs with minimal effort and zero UI work. Using this tool will help us diagnose and prevent potential issues before they become problems.

Additional Resources

Azure documentation
Custom Events in more detail
Detailed walkthrough
Official GitHub Repository

Disclaimer: The data used to generate graphs in this post is test data for the purpose of demonstration only and does not reflect production values.

This post is meant to get you up to speed with WebJobs as quickly as possible. It does not however contain every detail about this topic. I've included a few links at the end which contain much more detail.

Overview

A WebJob in the context of an Azure WebApp is a mechanism for running background processes. This can be useful in cases where a user completes a task via the WebApp which requires some processing, but that processing does not need to be carried out immediately. WebJobs are a good fit for any asynchronous task such as reacting to events, processing messages from queues, or performing maintenance work.

Why WebJobs?

Aren't they just glorified console apps? Not really. WebJobs provide several advantages over console apps. Some of these advantages are:

• Logging: just link your Azure Storage account and you'll get a decent logging dashboard in no time.
• Scaling: if your WebJob experiences high load, the AppService underneath it can be configured to AutoScale. No need to manually deploy and manage additional instances of your job to more virtual machines.
• REST API: programmatically control your WebJobs through a REST API. This is much easier than using RDP especially if you want to manage your WebJobs in code.
• WebJobs SDK: the SDK has many out of the box integrations with Azure Storage queues, Blob Storage, ServiceBus queues and topics, and many more. You can of course implement this on your own, but being able to get up and running quickly can be very helpful.
• Scheduling: WebJobs can be scheduled easily using several methods (more on this later)

When not to use WebJobs

If your code has a dependency on specific software that needs to be installed on the host virtual machine, then using WebJobs might not work for you. With WebJobs, you don't have direct control over the virtual machine they run on. Even in this case, I suggest finding Nuget packages that perform the same job instead of depending on other processes to be present on the host virtual machine.

Types of WebJobs

There are 3 types of WebJobs

• Triggered: runs on a schedule
• On Demand: run manually
• Continuous: runs continuously

Continuous WebJobs are a good fit for reacting to events such as processing messages from queues or reacting to file changes in Blob Storage..etc. Triggered and On Demand WebJobs are good for any background processing tasks.

Creating & Deploying WebJobs

A WebJob runs in the context of an Azure WebApp or API App. The WebApp acts as the container for the WebJob. If you have an existing Console App, you can zip and upload it through the Azure portal or deploy it through Visual Studio by right clicking on the project and selecting "Publish as Azure WebJob".

Best practices when creating a new WebJob:

• Add a new project to your solution. You can find the WebJob template under Cloud => Azure WebJob. Note: Make sure you specify the .NET version you want to use. (assuming you're using Visual Studio)

• Update nuget packages as necessary since the template has old verisons.
• If you want to deploy to multiple environments (staging, production..etc), add the SlowCheetah Nuget package (you will need that for transforming app configs).
• Add configuration transforms to your app.config. You can do that by right clicking on your app.config file and selecting the "Add Transform" option. Note: you will need the SlowCheetah Visual Studio extention for this option to be available.
• When you're ready to publish your WebJob, right click on it and choose "Publish as Azure WebJob". A publishing wizard will guide you through to choose the type and schedule of your WebJob.
• Once your WebJob is deployed to Azure, you can find it under the WebApp or API App it was deployed to by going to Settings => WebJobs.

Scheduling Options

There are three ways you can schedule your triggered WebJobs

• Azure Scheduler
• CRON timer expressions using the WebJobs SDK Extensions Nuget package
• CRON expression in a settings.job file in the root directory of the WebJob

If you decide to use CRON expressions, make sure the Always On option on the web app is turned on.

Note: if you need scheduling recurrence down to the minute level, you will need to upgrade the Azure Scheduler instance to the Standard tier.

Logs

WebJobs provide a dashboard where you can see all the jobs you have, their running status, and individual invocations along with any output they generate.

In order to access the logs for a WebJob, an Azure Storage Account needs to be linked with the WebJob. In App.config, there are two connection strings:

<add name="AzureWebJobsDashboard" connectionString="" />  
<add name="AzureWebJobsStorage" connectionString="" />  

You can populate these with the connection string for your Storage Account using this format:

DefaultEndpointsProtocol=https;AccountName=NAME;AccountKey=KEY  

Important: On the WebApp you deployed the WebJob to, go to Settings => Connection strings and set two connection strings with the same names and values you set in your App.config and set their type to "Custom". Once you do that, it will get picked up by convention and you'll be able to view the dashboard.

Tip: for simplicity, I like to use the same Storage Account for both Storage and Dashboard. I found it simple to have a single connection string in my App.config called "Azure Storage" and use that when initializing the WebJob.

var storageConnection = ConfigurationManager.ConnectionStrings["AzureStorage"].ConnectionString;  
var config = new JobHostConfiguration(storageConnection);  

var host = new JobHost(config);  
host.Start();  

This way, there is one less connection string to worry about and transform.

Logs Retention

You can control the history size and retention for triggered WebJobs by using the WEBJOBS_HISTORY_SIZE setting (default is 50 invocations). Unfortunately, there isn't a way to control the history size for continuous WebJobs yet. I've opened an issue on the WebJob's GitHub repository for that which will hopefully be included in a future release.

Tips & Tricks

• In the WebJob project under Properties, a webjob-publish-settings.json file is created which holds the configuration details such as the run mode and schedule. You can edit the schedule there or delete the file entirely to get the scheduling dialog in Visual Studio again.
• For continuous WebJobs, the Always On setting needs to be turned on in the WebApp's settings to prevent the WebApp from getting unloaded when idle.
• You can adjust the timeout value that the WebJobs SDK uses to kill idle function invocations (no CPU or IO activity). You can do that by adding an application setting WEBJOBS_IDLE_TIMEOUT with a value in seconds in the WebApp's application settings. This setting can be set programmatically as well.
• If you want to take a look at the binaries deployed to the Azure WebApp, connect to the WebApp via FTP using a client such as FileZilla, jobs will be under the following directory: site/wwwroot/app_data/jobs
• Enable Development Settings when running locally. This will adjust logging verbosity and other settings in order to optimize for local development. You can do that by:

if (config.IsDevelopment)  
{
    config.UseDevelopmentSettings();
}

Note: you'll need to add an environment variable to your machine with a name "AzureWebJobsEnv" and a value of "Development"

WebJobs & AppService AutoScale

Azure AppService (the underlying infrastructure that hosts the WebJob) can AutoScale to handle increasing load automatically (see here for more info). This will result in having multiple instances of you WebJob running. If you want to prevent your WebJob from AutoScaling, include a settings.job file with the following attributes:

{
  "is_singleton" : true
}

Note: make sure you right click on the file, go to properties, and change the "Copy to Output Directory" setting to "Copy Always" so this file gets included when the WebJob gets deployed.

REST API

You can interact with your WebJobs through a REST API. This comes handy when you want to start/stop or adjust the settings of your WebJobs programmatically. A good example of this is a SQL database maintenance WebJob we have that runs weekly to rebuild indexes and update statistics. That WebJob uses this API to stop all WebJobs that write to that database, performs its database maintenance work, then starts the jobs back up.

Note: you'll need to authenticate with the REST API via basic authentication. The username and password are found in the publish profile of the WebApp.

Additional Resources

Azure documentation
Introducing Windows Azure WebJobs - Scott Hansleman's blog post
Azure Websites and WebJobs - Pluralsight course
Sample projects