Azure Diagnostics

Diagnostics and monitoring for services and applications that don’t run in the cloud doesn’t require a special attention since you have physical  access to the production server, more or less, it depends on how you’ve decided to host them. But in the cloud there are several challenges with diagnostics like:

• Many instances
• They move around
• Massive amount of data
• Can’t remote desktop in
• No remote tools (yet)

So Microsoft have implemented a monitoring agent (MonAgentHost.exe) which runs on each instance in the cloud. The agent is started automatically by default. The listener is wired up in the app/web.config, like any other TraceListener. More you need to define a storage account connection string for this listener. In a nutshell it works in 5 steps:

1. 1.Role instance starts and by default the monitoring agent is started too.
2. 2.Diagnostic Monitor Starts
3. 3.Monitor must be configured at start time or remotely any time using service configuration file.
4. 4.Monitor starts to buffer data locally. The user might set a quota, too.
5. 5.User initiates transfer to storage. The transfer can be scheduled  or on demand. I recommend a scheduled transfer. In this way diagnostics storage is up to date with live locally data.
6. 

 

Below is the table with items that can be monitored in the cloud, if they are activated by default or not and the storage destination type.

Data Source	Default	Destination
Trace Logs	Enabled	Azure Table
Diagnostic Infrastructure Logs	Enabled	Azure Table
IIS Logs	Enabled	Blob
Performance Counters	Disabled	Azure Table
Windows Event Logs	Disabled	Azure Table
IIS Failed Request Logs	Disabled	Blob
Crash Dumps	Disabled	Blob
Arbitrary Files	Disabled	Blob

Implementing Azure Diagnostics:

First Diagnostic Agent is loaded as an Azure module the ServiceDefinition.csdef:

Then module expects a connection string:

Attention, a production connection string must be https! Like this:

<Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="DefaultEndpointsProtocol=https;

AccountName=YOURACCOUNT;AccountKey=YOURKEY" />

The common pattern to configure diagnostics suppose:

• Get Config
  • From default
  • Current running
• Make a change to the config
  • If changed within the instance it will affect only that instance. Don’t forget to start the agent immediately
  • If changed from outside for all roles then
    • change the central file
    • agent notices a change and reloads
    • affects all instances of the role
• Start the Diagnostics agent with the new configuration

The code:

Method 1:

• I’ve set up a transfer to the storage every one minute
• Also I’ve added “System” and “Application” transfer from Event Viewer

Code Snippet

1. //—————————————-method 1—————————-//
2.                 //get config
3.                 CloudStorageAccount storageAcc = CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString"));
4.                 RoleInstanceDiagnosticManager ridm = storageAcc.CreateRoleInstanceDiagnosticManager(RoleEnvironment.DeploymentId,
5.                     RoleEnvironment.CurrentRoleInstance.Role.Name, RoleEnvironment.CurrentRoleInstance.Id);
6.                 DiagnosticMonitorConfiguration dmc = ridm.GetCurrentConfiguration();
7.  
8.                 //change config
9.                 // Transfer logs to storage every minute
10.                 dmc.Logs.ScheduledTransferPeriod = tsTenSeconds;
11.                 dmc.DiagnosticInfrastructureLogs.ScheduledTransferPeriod = tsTenSeconds;
12.                 dmc.Directories.ScheduledTransferPeriod = tsTenSeconds;
13.                 dmc.WindowsEventLog.ScheduledTransferPeriod = tsTenSeconds;
14.                                 
15.                 dmc.WindowsEventLog.DataSources.Add("System!*");
16.                 dmc.WindowsEventLog.DataSources.Add("Application!*");
17.                 dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Information;
18.  
19.                 ridm.SetCurrentConfiguration(dmc);
20.                 //—————————————-end method 1—————————-//

Method 2:

• I’ve loaded the default configuration and I’ve set up a scheduled transfer

Code Snippet

1. //—————————————-method 2—————————-//
2.                 // Start up the diagnostic manager with the given configuration
3.                 DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();
4.                 dmc.Logs.ScheduledTransferPeriod = tsTenSeconds;
5.                 // Transfer verbose, critical, etc. logs
6.                 dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;
7.                 DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", dmc);
8.                 //—————————————-end method 2—————————-//

Method 3:

• I’ve created a custom listener that writes log entries in a dedicated Azure table

Code Snippet

1. //—————————————-method 3—————————-//
2.                 CloudStorageAccount storageAcc = CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString"));
3.                 CloudTableClient tableStorage = storageAcc.CreateCloudTableClient();
4.                 tableStorage.CreateTableIfNotExist(TableStorageTraceListener.DIAGNOSTICS_TABLE);
5.                 AzureDiagnostics.TableStorageTraceListener listener =
6.                  new AzureDiagnostics.TableStorageTraceListener("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString")
7.                  {
8.                      Name = "TableStorageTraceListener"
9.                  };
10.                 System.Diagnostics.Trace.Listeners.Add(listener);
11.                 System.Diagnostics.Trace.AutoFlush = true;
12.                 Trace.Listeners.Add(new DiagnosticMonitorTraceListener());
13.                 //—————————————-end method 3—————————-//

Full code available here.

Visualizing the data

You might use Azure Storage Explorer to explore diagnostics storage account or you can use Cerebrata’s Azure Diagnostics Manager which is a dedicated tool for diagnostics management. It will display a more friendly UI than the Storage Explorer:

Advertisement